PyTorch Developer Podcast

Why is PyTorch's build so g-dang complicated. How to avoid having to deal with cmake at all? And if you have to deal with cmake, what are the most important things to know? And if you were going to improve our cmake, how would you go about doing it...

Further reading.

• The official CMake documentation is a great help and well worth reading https://cmake.org/documentation
• If you work in torch/csrc chances are you'll need to edit this file https://github.com/pytorch/pytorch/blob/master/tools/build_variables.bzl

Liner notes.

• multiple build systems: cmake, buck, xplat buck, ovrsource buck, bazel
  • tools/build_variables.bzl is read from cmake! append_filelist
    • but not used uniformly for all components! (ouch!)
• mashed together ATen and Caffe2 build systems (e.g., main library libtorch_cpu is defined in caffe2/CMakeLists.txt)
• cmake: not very much syntax, "everything is a function". This means you can look up constructs relatively easily; e.g., even if() is a command
• the general cmake model: "set a bunch of variables, run a bunch of commands". cmake is VERY GREPPABLE
  • but not everything is in CMakeLists.txt; check *.cmake too
  • the directory structure makes no sense, you really need to grep.
    (doing a lot of set PARENT_SCOPE to propagate stuff)
  • renaming a file? grep for it
  • primary hazard of refactoring: need to make sure all the variables
    are setup at the new location
• many directories are not recursive glob, beware of adding new directories
• old school cmake: literally everything is stuffed in variables (CMAKE_CXX_FLAGS). new school cmake: attach things to targets, things propagate when you depend on targets (public/private dependencies)
• add_library: the most important thing
• don't randomly change things and pray: have hypotheses and test them

Why does PyTorch use code generation as part of its build process? Why doesn't it use C++ templates? What things is code generation used for? What are the pros/consof using code generation? What are some other ways to do the same things we currently do with code generation?

Further reading.

• Top level file for the new code generation pipeline https://github.com/pytorch/pytorch/blob/master/tools/codegen/gen.py
• Out of tree external backend code generation from Brian Hirsh: https://github.com/pytorch/xla/issues/2871
• Documentation for native_functions.yaml https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/README.md (have you seen this README before? Yes you've seen this README before. Imma post it again.)

Outline:

• High level: reduce the amount of code in PyTorch, easier to develop
• Strongly typed python
• Stuff we're using codegen for
  • Meta point: stuff c++ metaprogramming can't do
  • C++ apis (functions, methods on classes)
    • Especially for forwarding (operator dot doko)
    • Prototypes for c++ to implement
  • YAML files used by external frameworks for binding (accidental)
  • Python arg parsing
  • pyi generation
  • Autograd classes for saving saved data
  • Otherwise complicated constexpr computation (e.g., parsing JIT
    schema)
• Pros
  • Better surface syntax (native_functions.yaml, jit schema,
    derivatives.yaml)
  • Better error messages (template messages famously bad)
  • Easier to organize complicated code; esp nontrivial input
    data structure
  • Easier to debug by looking at generated code
• Con
  • Not as portable (template can be used by anyone)
  • Less good modeling for C++ type based metaprogramming (we've replicated a crappy version of C++ type system in our codegen)
• Counterpoints in the design space
  • C++ templates: just as efficient
  • Boxed fallback: simpler, less efficient
• Open question: can you have best of both worlds, e.g., with partially evaluated interpreters?

Why is autograd so complicated? What are the constraints and features that go into making it complicated? What's up with it being written in C++? What's with derivatives.yaml and code generation? What's going on with views and mutation? What's up with hooks and anomaly mode? What's reentrant execution? Why is it relevant to checkpointing? What's the distributed autograd engine?

Further reading.

• Autograd notes in the docs https://pytorch.org/docs/stable/notes/autograd.html
• derivatives.yaml https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml
• Paper on autograd engine in PyTorch https://openreview.net/pdf/25b8eee6c373d48b84e5e9c6e10e7cbbbce4ac73.pdf

What is __torch_function__? Why would I want to use it? What does it have to do with keeping extra metadata on Tensors or torch.fx? How is it implemented? Why is __torch_function__ a really popular way of extending functionality in PyTorch? What makes it different from the dispatcher extensibility mechanism? What are some downsides of it being written this way? What are we doing about it?

Further reading.

• __torch_function__ RFC: https://github.com/pytorch/rfcs/blob/master/RFC-0001-torch-function-for-methods.md
• One of the original GitHub issues tracking the overall design discussion https://github.com/pytorch/pytorch/issues/24015
• Documentation for using __torch_function__ https://pytorch.org/docs/stable/notes/extending.html#extending-torch