- The scale of "glue code" systems compared to actual ML code in mature production settings is surprising - often only 5% ML code with 95% glue. The potential for hidden feedback loops and unintended consequences of changes was also alarming. It's surprising how rapidly technical debt can accumulate without care for abstraction and dependencies in ML systems
Tl/dr key points from the paper contrasted against long-established software engineering principles:
- Encapsulation and modular design are time-tested principles for managing complexity and enabling change since the late 1960s. Machine learning models, through subtle erosion of abstraction boundaries, undermine these foundations.
- Loose coupling and separation of concerns are vital for maintenance and innovation. Yet models become tightly entangled through shared inputs/outputs, with unintended wide-ranging impacts of even small changes.
- Well-defined interfaces allow components to evolve independently. But model aspects like inputs and preprocessing logic undesirably "leak" between unrelated tasks, blurring component responsibilities.
- Testability demands isolated units with explicitly defined behaviors. However, emergent model interactions defeats this, as behaviors depend on complex system dynamics not just component logic.
- Reasoning locally is key to comprehension. But entanglement incentivizes haphazard copying of code between unrelated tasks instead of principled design.
- Minimizing dependencies limits downstream ripple effects of changes. Yet data dependencies in particular propagate silently and are poorly traced compared to code links.
- Stable abstract layers support understanding at different granularities. Erosion of these abstraction layers impairs understanding of increasingly tangled systems.
Additional examples of how machine learning models can subtly erode abstraction boundaries:
- Inputs designed for one model may start being reused as inputs to other unrelated models, tightly coupling the systems in unexpected ways. For example, if a click-through prediction model's output became an input to a recommendation model.
- Features engineered for one problem domain may start being used for another different domain, even if the correlation doesn't reflect a true causal relationship. This can obscure the problem each model is actually solving.
- Internal representations or hidden layers of one model may leak out and be utilized by other parts of the system, whereas they were intended just for that model's internal use. This spreads the effects of any one model.
- Models trained on certain data distributions may start being used for slightly different prediction problems without considering how the shift in data may impact results, essentially blurring the boundaries of what problem each model solves.
- The processing or normalization steps used for one model's features may start being shared by other models without consideration of whether those steps are truly necessary for other unrelated problems.
- Model hyperparameters like learning rates or regularization constants optimized for one use case may get reused in other contexts without evaluating their appropriateness, decreasing encapsulation.
IYH two points
a) Usenix CSET 2016 Prelim study link accrued technical debt & software vulnerabilities https://www.usenix.org/system/files/conference/cset16/cset16-paper-nord.pdf
https://web.archive.org/web/20210413133511/https://twitter.com/daniel_bilar/status/777848690975342593
b) Most alarming finding
- The scale of "glue code" systems compared to actual ML code in mature production settings is surprising - often only 5% ML code with 95% glue. The potential for hidden feedback loops and unintended consequences of changes was also alarming. It's surprising how rapidly technical debt can accumulate without care for abstraction and dependencies in ML systems
Tl/dr key points from the paper contrasted against long-established software engineering principles:
- Encapsulation and modular design are time-tested principles for managing complexity and enabling change since the late 1960s. Machine learning models, through subtle erosion of abstraction boundaries, undermine these foundations.
- Loose coupling and separation of concerns are vital for maintenance and innovation. Yet models become tightly entangled through shared inputs/outputs, with unintended wide-ranging impacts of even small changes.
- Well-defined interfaces allow components to evolve independently. But model aspects like inputs and preprocessing logic undesirably "leak" between unrelated tasks, blurring component responsibilities.
- Testability demands isolated units with explicitly defined behaviors. However, emergent model interactions defeats this, as behaviors depend on complex system dynamics not just component logic.
- Reasoning locally is key to comprehension. But entanglement incentivizes haphazard copying of code between unrelated tasks instead of principled design.
- Minimizing dependencies limits downstream ripple effects of changes. Yet data dependencies in particular propagate silently and are poorly traced compared to code links.
- Stable abstract layers support understanding at different granularities. Erosion of these abstraction layers impairs understanding of increasingly tangled systems.
Additional examples of how machine learning models can subtly erode abstraction boundaries:
- Inputs designed for one model may start being reused as inputs to other unrelated models, tightly coupling the systems in unexpected ways. For example, if a click-through prediction model's output became an input to a recommendation model.
- Features engineered for one problem domain may start being used for another different domain, even if the correlation doesn't reflect a true causal relationship. This can obscure the problem each model is actually solving.
- Internal representations or hidden layers of one model may leak out and be utilized by other parts of the system, whereas they were intended just for that model's internal use. This spreads the effects of any one model.
- Models trained on certain data distributions may start being used for slightly different prediction problems without considering how the shift in data may impact results, essentially blurring the boundaries of what problem each model solves.
- The processing or normalization steps used for one model's features may start being shared by other models without consideration of whether those steps are truly necessary for other unrelated problems.
- Model hyperparameters like learning rates or regularization constants optimized for one use case may get reused in other contexts without evaluating their appropriateness, decreasing encapsulation.