| commit | be98c5d12d2392a18eae15f1101457adbec0f876 | [log] [tgz] |
|---|---|---|
| author | Sam Gross <colesbury@gmail.com> | Fri Dec 30 01:21:34 2016 -0500 |
| committer | Soumith Chintala <soumith@gmail.com> | Fri Dec 30 01:21:34 2016 -0500 |
| tree | 0b52f969e281ec17d4f0cbde2f28900cd740f17f | |
| parent | bc6a71b1f5f0e814f8be5bcc1f4c05e84bb24373 [diff] |
Start documenting torch.Tensor (#377)
| Python | Linux CPU | Linux GPU |
|---|---|---|
| 2.7.8 |  | |
| 2.7 | | |
| 3.5 | | |
| Nightly | |
The project is still under active development and is likely to drastically change in short periods of time. We will be announcing API changes and important developments via a newsletter, github issues and post a link to the issues on slack. Please remember that at this stage, this is an invite-only closed alpha, and please don't distribute code further. This is done so that we can control development tightly and rapidly during the initial phases with feedback from you.
PyTorch is a python package with the goal of providing GPU-optimized Tensor computation and deep learning. You can reuse your favorite python packages such as numpy, scipy and Cython to extend PyTorch to your own needs, or use the simple extension API that we provide.
At a granular level, PyTorch is a library that consists of the following components:
| _ | _ |
|---|---|
| torch | a Tensor library like NumPy, with strong GPU support |
| torch.autograd | a tape based automatic differentiation library that supports all differentiable Tensor operations in torch |
| torch.nn | a neural networks library deeply integrated with autograd designed for maximum flexibility |
| torch.optim | an optimization package to be used with torch.nn with standard optimization methods such as SGD, RMSProp, LBFGS, Adam etc. |
| torch.multiprocessing | python multiprocessing, but with magical memory sharing of torch Tensors across processes. Useful for data loading and hogwild training. |
| torch.utils | DataLoader, Trainer and other utility functions for convenience |
| torch.legacy(.nn/.optim) | legacy code that has been ported over from torch for backward compatibility reasons |
Usually one uses PyTorch either as:
Elaborating further:
If you use numpy, then you have used Tensors (a.k.a ndarray).
PyTorch provides Tensors that can live either on the CPU or the GPU, and accelerate compute by a huge amount.
We provide 300+ tensor routines to accelerate and fit your scientific computation needs.
And they are fast!
PyTorch has a unique way of building neural networks: using and replaying a tape recorder.
Most frameworks such as TensorFlow, Theano, Caffe and CNTK have a static view of the world. One has to build a neural network, and reuse the same structure again and again. Changing the way the network behaves means that one has to start from scratch.
With PyTorch, we use a technique called Reverse-mode auto-differentiation, which allows you to change the way your network behaves arbitrarily with zero lag or overhead. Our inspiration comes from several research papers on this topic, as well as current and past work such as autograd, autograd, Chainer, etc.
While this technique is not unique to PyTorch, it's definitely the fastest implementation of it.
You get the best of speed and flexibility for your crazy research.
PyTorch is not a Python binding into a monolothic C++ framework.
It is built to be deeply integrated into Python.
You can use it naturally like you would use numpy / scipy / scikit-learn etc.
You can write your new neural network layers in Python itself, using your favorite libraries and use packages such as Cython and Numba.
We dont want to reinvent the wheel, we want to reuse all the wheels that have been built.
PyTorch is designed to be intuitive, linear in thought and easy to use. When you execute a line of code, it gets executed. There isn't an asynchronous view of the world. When you drop into a debugger, or receive error messages and stack traces, understanding them is straight-forward, as and easy to understand. The stack-trace points to exactly where your code was defined. We hope you never spend hours debugging your code because of bad stack traces or asynchronous and opaque execution engines.
PyTorch is as fast as the fastest deep learning framework out there. We integrate acceleration frameworks such as Intel MKL and NVIDIA (CuDNN, NCCL) for maximum speed. You can use multiple GPUs and machines with maximum efficiency.
The memory usage in PyTorch is extremely efficient. We've written custom memory allocators for the GPU to make sure that your deep learning models are maximally memory efficient. This enables you to train bigger deep learning models than before.
Writing new neural network modules, or interfacing with PyTorch‘s Tensor API is a breeze, thanks to an easy to use extension API that is efficient and easy to use. Writing C or Cython functions to add new neural network modules is straight-forward and painless. At the core, all the value of PyTorch -- it’s CPU and GPU Tensor and NeuralNet backends -- are written in simple libraries with a C99 API. They are mature and have been tested for years.
conda install pytorch -c https://conda.anaconda.org/t/6N-MsQ4WZ7jo/soumith
export CMAKE_PREFIX_PATH=[anaconda root directory] conda install numpy mkl conda install -c soumith magma-cuda75# or magma-cuda80
export MACOSX_DEPLOYMENT_TARGET=10.9 # for OSX pip install -r requirements.txt python setup.py install
Three pointers to get you started:
We will run the alpha releases weekly for 6 weeks. After that, we will reevaluate progress, and if we are ready, we will hit beta-0. If not, we will do another two weeks of alpha.
The beta phases will be leaning more towards working with all of you, convering your use-cases, active development on non-core aspects.
We‘ve decided that it’s time to rewrite/update parts of the old torch API, even if it means losing some of backward compatibility.
This tutorial takes you through the biggest changes and walks you through PyTorch
For brevity,
To reiterate, we recommend that you go through This tutorial
Pickling tensors is supported, but requires making a temporary copy of all data in memory and breaks sharing.
For this reason we're providing torch.load and torch.save, that are free of these problems.
They have the same interfaces as pickle.load (file object) and pickle.dump (serialized object, file object) respectively.
For now the only requirement is that the file should have a fileno method, which returns a file descriptor number (this is already implemented by objects returned by open).
Objects are serialized in a tar archive consisting of four files:
sys_info - protocol version, byte order, long size, etc.pickle - pickled objecttensors - tensor metadatastorages - serialized dataWe made PyTorch to seamlessly integrate with python multiprocessing. What we've added specially in torch.multiprocessing is the seamless ability to efficiently share and send tensors over from one process to another. (technical details of implementation) This is very useful for example in:
Here are a couple of examples for torch.multiprocessing
# loaders.py # Functions from this file run in the workers def fill(queue): while True: tensor = queue.get() tensor.fill_(10) queue.put(tensor) def fill_pool(tensor): tensor.fill_(10)
# Example 1: Using multiple persistent processes and a Queue # process.py import torch import torch.multiprocessing as multiprocessing from loaders import fill # torch.multiprocessing.Queue automatically moves Tensor data to shared memory # So the main process and worker share the data queue = multiprocessing.Queue() buffers = [torch.Tensor(2, 2) for i in range(4)] for b in buffers: queue.put(b) processes = [multiprocessing.Process(target=fill, args=(queue,)).start() for i in range(10)]
# Example 2: Using a process pool # pool.py import torch from torch.multiprocessing import Pool from loaders import fill_pool tensors = [torch.Tensor(2, 2) for i in range(100)] pool = Pool(10) pool.map(fill_pool, tensors)