| commit | 0b52b3c79dacd6a0ce41bca1cb727bf2ba4fd161 | [log] [tgz] |
|---|---|---|
| author | Aapo Kyrola <akyrola@fb.com> | Tue Dec 13 15:15:36 2016 -0800 |
| committer | Bram Wasti <bwasti@dev11999.prn1.facebook.com> | Thu Dec 15 12:01:30 2016 -0800 |
| tree | 6e03d90b4eafb03d5e95cd774f7b1760dc74b315 | |
| parent | 4858a6bc6fcd71d9af4ce956a3b6a58b01889fd2 [diff] |
Generalize threaded data input via queues + Everstore input Summary: Xray sampler (originally by ajtulloch) and prigoyal's resnet trainer use variants of the threaded data input where worker threads put stuff into a python queue that is drained by an enqueuer thread that dumps those batches to a Caffe2 queue, that is then drained by the net's DequeueBlobs operator. There is a lot of boilerplate, which is also quite complicated. This diff is an attempt to generalize that general stuff under a new module "data_workers" (name could be improved). Basically you pass it a function that is able to return chunks of data (usually data + labels). I also created a module 'everstore_data_input' which generalizes everstore-origin data input with preprocessing function (image augmentation , for example). See how I refactored sampler.py for the usage. Next we could create fetcher function for Laser data. Differential Revision: D4297667 fbshipit-source-id: 8d8a863b177784ae13940730a27dc76cd1dd3dac
Caffe2 is a deep learning framework made with expression, speed, and modularity in mind. It is an experimental refactoring of Caffe, and allows a more flexible way to organize computation.
Read the installation instructions for installation details.
Caffe2 is released under the BSD 2-Clause license.