Hogwild Examples

Hogwild! includes example implementations for the following tasks:

• Sparse SVM
• Multicut
• Maxcut (tracenorm)

SVM Example

Hogwild! can train an SVM using parallel incremental gradient descent. This example uses 3 threads to train the SVM. The stepsize and step decay are paremters for the gradient decsient algorithm. For improved performance on large datasets, Hogwild! can load binary format files. Standard TSV files can be converted to/from the binary formating using the provided convert or unconvert tools provided in the release.

$ bin/svm --splits 3 --stepinitial 0.1 --step_decay 0.8 --mu 1 --binary 1 \
          data/RCV1.train.bin data/RCV1.test.tsv 
Loading binary file...
Loaded binary file!
Loaded 47236 examples
wall_clock: 1.09927    Going Hogwild!
epoch: 1 wall_clock: 1.12122 train_time: 0.01427 test_time: 0.00367 epoch_time: 0.01427 
    train_rmse: 0.38081 test_rmse: 0.38081
epoch: 2 wall_clock: 1.14172 train_time: 0.02760 test_time: 0.00353 epoch_time: 0.01333 
    train_rmse: 0.33954 test_rmse: 0.33954
epoch: 3 wall_clock: 1.16196 train_time: 0.04071 test_time: 0.00357 epoch_time: 0.01311 
    train_rmse: 0.31872 test_rmse: 0.31872
epoch: 4 wall_clock: 1.18223 train_time: 0.05397 test_time: 0.00352 epoch_time: 0.01326 
    train_rmse: 0.30506 test_rmse: 0.30506
epoch: 5 wall_clock: 1.20237 train_time: 0.06702 test_time: 0.00351 epoch_time: 0.01305 
    train_rmse: 0.29445 test_rmse: 0.29445

Best Ball Maxcut (Tracenorm)

This is a tracenorm matrix factorization example. Here, the "Best Ball" approach allows for the gradient descent algorithm to be auto tuned. Instead of having to select a single stepsize, multiple parameters are run concurrently. Multiple models are created and trained concurrently as Hogwild! scans over the data. After each pass over the data (each "epoch"), Hogwild! computes the train and test RMSE for each model or "ball." The model with the lowest harmonic mean of train and test RMSE is selected as the starting model for the next epoch. This means at each epoch, all models start at the best solution that has been found so far. Even if some parameters diverge, such as ball #0 in epoch 4 below, Hogwild! will still converge.

Also, notice that '--file_scan 2.5' instructs Hogwild! to only load at most 2.5 GB of the file into memory at one time; this allows Hogwild! to run over datasets that do not fit in memory.

bin/bbtracenorm --mu params/mus.lst --maxrank 30 --epochs 8 \
    --stepinitial params/steps.lst --step_decay 0.95 --splits 1 \
    --matlab-tsv 1 --file_scan 2.5 \
    data/netflix.train.bin data/netflix.test.tsv 
Loaded 281702 test examples.
Loaded 100198806 train examples.
number of rows: 480189
number of cols: 17770
Ball #0 = (step=0.100000, mu=0.010000)
Ball #1 = (step=0.010000, mu=0.010000)
Ball #2 = (step=0.001000, mu=0.010000)
Mean = 3.604093
wall_clock: 4.91098    Going Hogwild!
Ball #0 harmonic mean of RMSEs = 1.012457
Ball #1 harmonic mean of RMSEs = 0.971785
Ball #2 harmonic mean of RMSEs = 1.106682
epoch: 1 wall_clock: 111.49881 best_ball: 1 harm_mean: 0.97178 
    train_rmse: 0.93596 test_rmse: 1.01046
Ball #0 harmonic mean of RMSEs = 0.978227
Ball #1 harmonic mean of RMSEs = 0.889597
Ball #2 harmonic mean of RMSEs = 0.931522
epoch: 2 wall_clock: 207.54638 best_ball: 1 harm_mean: 0.88960 
    train_rmse: 0.83836 test_rmse: 0.94750
Ball #0 harmonic mean of RMSEs = 0.968183
Ball #1 harmonic mean of RMSEs = 0.859980
Ball #2 harmonic mean of RMSEs = 0.878059
epoch: 3 wall_clock: 302.53194 best_ball: 1 harm_mean: 0.85998 
    train_rmse: 0.79979 test_rmse: 0.92997
Ball #0 harmonic mean of RMSEs = -nan
Ball #1 harmonic mean of RMSEs = 0.843802
Ball #2 harmonic mean of RMSEs = 0.850561
epoch: 4 wall_clock: 397.64292 best_ball: 1 harm_mean: 0.84380 
    train_rmse: 0.77671 test_rmse: 0.92359