Note
Click here to download the full example code
Stochastic Training of GNN for Link Prediction¶
This tutorial will show how to train a multi-layer GraphSAGE for link
prediction on ogbn-arxiv provided by Open Graph Benchmark
(OGB). The dataset
contains around 170 thousand nodes and 1 million edges.
By the end of this tutorial, you will be able to
Train a GNN model for link prediction on a single GPU with DGL’s neighbor sampling components.
This tutorial assumes that you have read the Introduction of Neighbor Sampling for GNN Training and Neighbor Sampling for Node Classification.
Link Prediction Overview¶
Link prediction requires the model to predict the probability of existence of an edge. This tutorial does so by computing a dot product between the representations of both incident nodes.
\[\hat{y}_{u\sim v} = \sigma(h_u^T h_v)\]
It then minimizes the following binary cross entropy loss.
\[\mathcal{L} = -\sum_{u\sim v\in \mathcal{D}}\left( y_{u\sim v}\log(\hat{y}_{u\sim v}) + (1-y_{u\sim v})\log(1-\hat{y}_{u\sim v})) \right)\]
This is identical to the link prediction formulation in the previous tutorial on link prediction.
Loading Dataset¶
This tutorial loads the dataset from the ogb package as in the
previous tutorial.
import dgl
import torch
import numpy as np
from ogb.nodeproppred import DglNodePropPredDataset
dataset = DglNodePropPredDataset('ogbn-arxiv')
device = 'cpu' # change to 'cuda' for GPU
graph, node_labels = dataset[0]
# Add reverse edges since ogbn-arxiv is unidirectional.
graph = dgl.add_reverse_edges(graph)
print(graph)
print(node_labels)
node_features = graph.ndata['feat']
node_labels = node_labels[:, 0]
num_features = node_features.shape[1]
num_classes = (node_labels.max() + 1).item()
print('Number of classes:', num_classes)
idx_split = dataset.get_idx_split()
train_nids = idx_split['train']
valid_nids = idx_split['valid']
test_nids = idx_split['test']
Out:
WARNING:root:The OGB package is out of date. Your version is 1.2.4, while the latest version is 1.3.2.
Graph(num_nodes=169343, num_edges=2332486,
ndata_schemes={'year': Scheme(shape=(1,), dtype=torch.int64), 'feat': Scheme(shape=(128,), dtype=torch.float32)}
edata_schemes={})
tensor([[ 4],
[ 5],
[28],
...,
[10],
[ 4],
[ 1]])
Number of classes: 40
Defining Neighbor Sampler and Data Loader in DGL¶
Different from the link prediction tutorial for full graph, a common practice to train GNN on large graphs is to iterate over the edges in minibatches, since computing the probability of all edges is usually impossible. For each minibatch of edges, you compute the output representation of their incident nodes using neighbor sampling and GNN, in a similar fashion introduced in the large-scale node classification tutorial.
DGL provides dgl.dataloading.EdgeDataLoader to
iterate over edges for edge classification or link prediction tasks.
To perform link prediction, you need to specify a negative sampler. DGL
provides builtin negative samplers such as
dgl.dataloading.negative_sampler.Uniform. Here this tutorial uniformly
draws 5 negative examples per positive example.
negative_sampler = dgl.dataloading.negative_sampler.Uniform(5)
After defining the negative sampler, one can then define the edge data
loader with neighbor sampling. To create an EdgeDataLoader for
link prediction, provide a neighbor sampler object as well as the negative
sampler object created above.
sampler = dgl.dataloading.MultiLayerNeighborSampler([4, 4])
train_dataloader = dgl.dataloading.EdgeDataLoader(
# The following arguments are specific to NodeDataLoader.
graph, # The graph
torch.arange(graph.number_of_edges()), # The edges to iterate over
sampler, # The neighbor sampler
negative_sampler=negative_sampler, # The negative sampler
device=device, # Put the MFGs on CPU or GPU
# The following arguments are inherited from PyTorch DataLoader.
batch_size=1024, # Batch size
shuffle=True, # Whether to shuffle the nodes for every epoch
drop_last=False, # Whether to drop the last incomplete batch
num_workers=0 # Number of sampler processes
)
You can peek one minibatch from train_dataloader and see what it
will give you.
input_nodes, pos_graph, neg_graph, mfgs = next(iter(train_dataloader))
print('Number of input nodes:', len(input_nodes))
print('Positive graph # nodes:', pos_graph.number_of_nodes(), '# edges:', pos_graph.number_of_edges())
print('Negative graph # nodes:', neg_graph.number_of_nodes(), '# edges:', neg_graph.number_of_edges())
print(mfgs)
Out:
Number of input nodes: 57189
Positive graph # nodes: 6905 # edges: 1024
Negative graph # nodes: 6905 # edges: 5120
[Block(num_src_nodes=57189, num_dst_nodes=23942, num_edges=88948), Block(num_src_nodes=23942, num_dst_nodes=6905, num_edges=24144)]
The example minibatch consists of four elements.
The first element is an ID tensor for the input nodes, i.e., nodes whose input features are needed on the first GNN layer for this minibatch.
The second element and the third element are the positive graph and the negative graph for this minibatch. The concept of positive and negative graphs have been introduced in the full-graph link prediction tutorial. In minibatch training, the positive graph and the negative graph only contain nodes necessary for computing the pair-wise scores of positive and negative examples in the current minibatch.
The last element is a list of MFGs storing the computation dependencies for each GNN layer. The MFGs are used to compute the GNN outputs of the nodes involved in positive/negative graph.
Defining Model for Node Representation¶
The model is almost identical to the one in the node classification tutorial. The only difference is that since you are doing link prediction, the output dimension will not be the number of classes in the dataset.
import torch.nn as nn
import torch.nn.functional as F
from dgl.nn import SAGEConv
class Model(nn.Module):
def __init__(self, in_feats, h_feats):
super(Model, self).__init__()
self.conv1 = SAGEConv(in_feats, h_feats, aggregator_type='mean')
self.conv2 = SAGEConv(h_feats, h_feats, aggregator_type='mean')
self.h_feats = h_feats
def forward(self, mfgs, x):
h_dst = x[:mfgs[0].num_dst_nodes()]
h = self.conv1(mfgs[0], (x, h_dst))
h = F.relu(h)
h_dst = h[:mfgs[1].num_dst_nodes()]
h = self.conv2(mfgs[1], (h, h_dst))
return h
model = Model(num_features, 128).to(device)
Defining the Score Predictor for Edges¶
After getting the node representation necessary for the minibatch, the last thing to do is to predict the score of the edges and non-existent edges in the sampled minibatch.
The following score predictor, copied from the link prediction tutorial, takes a dot product between the incident nodes’ representations.
import dgl.function as fn
class DotPredictor(nn.Module):
def forward(self, g, h):
with g.local_scope():
g.ndata['h'] = h
# Compute a new edge feature named 'score' by a dot-product between the
# source node feature 'h' and destination node feature 'h'.
g.apply_edges(fn.u_dot_v('h', 'h', 'score'))
# u_dot_v returns a 1-element vector for each edge so you need to squeeze it.
return g.edata['score'][:, 0]
Evaluating Performance (Optional)¶
There are various ways to evaluate the performance of link prediction. This tutorial follows the practice of GraphSAGE paper, where it treats the node embeddings learned by link prediction via training and evaluating a linear classifier on top of the learned node embeddings.
To obtain the representations of all the nodes, this tutorial uses neighbor sampling as introduced in the node classification tutorial.
Note
If you would like to obtain node representations without neighbor sampling during inference, please refer to this user guide.
def inference(model, graph, node_features):
with torch.no_grad():
nodes = torch.arange(graph.number_of_nodes())
sampler = dgl.dataloading.MultiLayerNeighborSampler([4, 4])
train_dataloader = dgl.dataloading.NodeDataLoader(
graph, torch.arange(graph.number_of_nodes()), sampler,
batch_size=1024,
shuffle=False,
drop_last=False,
num_workers=4,
device=device)
result = []
for input_nodes, output_nodes, mfgs in train_dataloader:
# feature copy from CPU to GPU takes place here
inputs = mfgs[0].srcdata['feat']
result.append(model(mfgs, inputs))
return torch.cat(result)
import sklearn.metrics
def evaluate(emb, label, train_nids, valid_nids, test_nids):
classifier = nn.Linear(emb.shape[1], num_classes).to(device)
opt = torch.optim.LBFGS(classifier.parameters())
def compute_loss():
pred = classifier(emb[train_nids].to(device))
loss = F.cross_entropy(pred, label[train_nids].to(device))
return loss
def closure():
loss = compute_loss()
opt.zero_grad()
loss.backward()
return loss
prev_loss = float('inf')
for i in range(1000):
opt.step(closure)
with torch.no_grad():
loss = compute_loss().item()
if np.abs(loss - prev_loss) < 1e-4:
print('Converges at iteration', i)
break
else:
prev_loss = loss
with torch.no_grad():
pred = classifier(emb.to(device)).cpu()
label = label
valid_acc = sklearn.metrics.accuracy_score(label[valid_nids].numpy(), pred[valid_nids].numpy().argmax(1))
test_acc = sklearn.metrics.accuracy_score(label[test_nids].numpy(), pred[test_nids].numpy().argmax(1))
return valid_acc, test_acc
Defining Training Loop¶
The following initializes the model and defines the optimizer.
The following is the training loop for link prediction and evaluation, and also saves the model that performs the best on the validation set:
import tqdm
import sklearn.metrics
best_accuracy = 0
best_model_path = 'model.pt'
for epoch in range(1):
with tqdm.tqdm(train_dataloader) as tq:
for step, (input_nodes, pos_graph, neg_graph, mfgs) in enumerate(tq):
# feature copy from CPU to GPU takes place here
inputs = mfgs[0].srcdata['feat']
outputs = model(mfgs, inputs)
pos_score = predictor(pos_graph, outputs)
neg_score = predictor(neg_graph, outputs)
score = torch.cat([pos_score, neg_score])
label = torch.cat([torch.ones_like(pos_score), torch.zeros_like(neg_score)])
loss = F.binary_cross_entropy_with_logits(score, label)
opt.zero_grad()
loss.backward()
opt.step()
tq.set_postfix({'loss': '%.03f' % loss.item()}, refresh=False)
if (step + 1) % 500 == 0:
model.eval()
emb = inference(model, graph, node_features)
valid_acc, test_acc = evaluate(emb, node_labels, train_nids, valid_nids, test_nids)
print('Epoch {} Validation Accuracy {} Test Accuracy {}'.format(epoch, valid_acc, test_acc))
if best_accuracy < valid_acc:
best_accuracy = valid_acc
torch.save(model.state_dict(), best_model_path)
model.train()
# Note that this tutorial do not train the whole model to the end.
break
Out:
0%| | 0/2278 [00:00<?, ?it/s]
0%| | 1/2278 [00:00<12:01, 3.16it/s, loss=28.579]
0%| | 2/2278 [00:00<10:01, 3.79it/s, loss=20.003]
0%| | 3/2278 [00:00<08:33, 4.43it/s, loss=13.889]
0%| | 4/2278 [00:00<07:30, 5.05it/s, loss=9.506]
0%| | 5/2278 [00:00<06:56, 5.46it/s, loss=6.420]
0%| | 6/2278 [00:01<06:32, 5.79it/s, loss=4.426]
0%| | 7/2278 [00:01<06:05, 6.21it/s, loss=3.140]
0%| | 8/2278 [00:01<05:47, 6.54it/s, loss=2.365]
0%| | 9/2278 [00:01<05:33, 6.79it/s, loss=1.922]
0%| | 10/2278 [00:01<05:23, 7.00it/s, loss=1.614]
0%| | 11/2278 [00:01<05:36, 6.74it/s, loss=1.426]
1%| | 12/2278 [00:01<05:25, 6.96it/s, loss=1.334]
1%| | 13/2278 [00:01<05:26, 6.94it/s, loss=1.236]
1%| | 14/2278 [00:02<05:17, 7.13it/s, loss=1.164]
1%| | 15/2278 [00:02<05:11, 7.26it/s, loss=1.097]
1%| | 16/2278 [00:02<05:07, 7.35it/s, loss=1.016]
1%| | 17/2278 [00:02<05:05, 7.40it/s, loss=0.970]
1%| | 18/2278 [00:02<05:04, 7.43it/s, loss=0.916]
1%| | 19/2278 [00:02<05:02, 7.46it/s, loss=0.883]
1%| | 20/2278 [00:02<05:04, 7.42it/s, loss=0.855]
1%| | 21/2278 [00:03<05:06, 7.36it/s, loss=0.834]
1%| | 22/2278 [00:03<05:05, 7.37it/s, loss=0.817]
1%|1 | 23/2278 [00:03<05:04, 7.40it/s, loss=0.807]
1%|1 | 24/2278 [00:03<05:03, 7.42it/s, loss=0.782]
1%|1 | 25/2278 [00:03<05:04, 7.39it/s, loss=0.781]
1%|1 | 26/2278 [00:03<05:02, 7.43it/s, loss=0.774]
1%|1 | 27/2278 [00:03<05:01, 7.46it/s, loss=0.768]
1%|1 | 28/2278 [00:04<05:08, 7.30it/s, loss=0.758]
1%|1 | 29/2278 [00:04<05:06, 7.34it/s, loss=0.754]
1%|1 | 30/2278 [00:04<05:03, 7.40it/s, loss=0.746]
1%|1 | 31/2278 [00:04<05:02, 7.44it/s, loss=0.740]
1%|1 | 32/2278 [00:04<05:00, 7.48it/s, loss=0.733]
1%|1 | 33/2278 [00:04<04:59, 7.50it/s, loss=0.732]
1%|1 | 34/2278 [00:04<04:59, 7.49it/s, loss=0.725]
2%|1 | 35/2278 [00:04<04:59, 7.48it/s, loss=0.731]
2%|1 | 36/2278 [00:05<05:08, 7.27it/s, loss=0.719]
2%|1 | 37/2278 [00:05<05:05, 7.34it/s, loss=0.715]
2%|1 | 38/2278 [00:05<05:03, 7.38it/s, loss=0.711]
2%|1 | 39/2278 [00:05<05:01, 7.43it/s, loss=0.707]
2%|1 | 40/2278 [00:05<04:59, 7.48it/s, loss=0.712]
2%|1 | 41/2278 [00:05<05:00, 7.45it/s, loss=0.706]
2%|1 | 42/2278 [00:05<05:00, 7.45it/s, loss=0.703]
2%|1 | 43/2278 [00:06<05:02, 7.39it/s, loss=0.703]
2%|1 | 44/2278 [00:06<05:03, 7.36it/s, loss=0.695]
2%|1 | 45/2278 [00:06<05:01, 7.41it/s, loss=0.700]
2%|2 | 46/2278 [00:06<04:59, 7.45it/s, loss=0.696]
2%|2 | 47/2278 [00:06<04:59, 7.44it/s, loss=0.693]
2%|2 | 48/2278 [00:06<04:59, 7.46it/s, loss=0.689]
2%|2 | 49/2278 [00:06<05:00, 7.42it/s, loss=0.696]
2%|2 | 50/2278 [00:06<05:01, 7.39it/s, loss=0.696]
2%|2 | 51/2278 [00:07<05:09, 7.19it/s, loss=0.694]
2%|2 | 52/2278 [00:07<05:05, 7.29it/s, loss=0.681]
2%|2 | 53/2278 [00:07<05:02, 7.36it/s, loss=0.690]
2%|2 | 54/2278 [00:07<05:02, 7.36it/s, loss=0.689]
2%|2 | 55/2278 [00:07<05:00, 7.41it/s, loss=0.681]
2%|2 | 56/2278 [00:07<04:58, 7.44it/s, loss=0.690]
3%|2 | 57/2278 [00:07<05:01, 7.37it/s, loss=0.686]
3%|2 | 58/2278 [00:08<05:00, 7.39it/s, loss=0.689]
3%|2 | 59/2278 [00:08<05:11, 7.13it/s, loss=0.681]
3%|2 | 60/2278 [00:08<05:06, 7.23it/s, loss=0.677]
3%|2 | 61/2278 [00:08<05:03, 7.31it/s, loss=0.681]
3%|2 | 62/2278 [00:08<04:59, 7.39it/s, loss=0.684]
3%|2 | 63/2278 [00:08<04:57, 7.44it/s, loss=0.678]
3%|2 | 64/2278 [00:08<04:56, 7.48it/s, loss=0.682]
3%|2 | 65/2278 [00:09<04:54, 7.51it/s, loss=0.680]
3%|2 | 66/2278 [00:09<04:54, 7.51it/s, loss=0.678]
3%|2 | 67/2278 [00:09<05:02, 7.31it/s, loss=0.679]
3%|2 | 68/2278 [00:09<05:01, 7.32it/s, loss=0.675]
3%|3 | 69/2278 [00:09<04:59, 7.38it/s, loss=0.678]
3%|3 | 70/2278 [00:09<04:56, 7.44it/s, loss=0.674]
3%|3 | 71/2278 [00:09<04:58, 7.40it/s, loss=0.681]
3%|3 | 72/2278 [00:09<04:56, 7.43it/s, loss=0.681]
3%|3 | 73/2278 [00:10<04:54, 7.48it/s, loss=0.677]
3%|3 | 74/2278 [00:10<05:02, 7.29it/s, loss=0.675]
3%|3 | 75/2278 [00:10<05:05, 7.20it/s, loss=0.675]
3%|3 | 76/2278 [00:10<05:03, 7.26it/s, loss=0.676]
3%|3 | 77/2278 [00:10<05:00, 7.33it/s, loss=0.674]
3%|3 | 78/2278 [00:10<05:02, 7.27it/s, loss=0.673]
3%|3 | 79/2278 [00:10<04:59, 7.34it/s, loss=0.674]
4%|3 | 80/2278 [00:11<04:56, 7.41it/s, loss=0.676]
4%|3 | 81/2278 [00:11<04:57, 7.39it/s, loss=0.674]
4%|3 | 82/2278 [00:11<05:07, 7.13it/s, loss=0.677]
4%|3 | 83/2278 [00:11<05:02, 7.25it/s, loss=0.671]
4%|3 | 84/2278 [00:11<05:05, 7.18it/s, loss=0.671]
4%|3 | 85/2278 [00:11<05:25, 6.74it/s, loss=0.676]
4%|3 | 86/2278 [00:11<05:16, 6.92it/s, loss=0.669]
4%|3 | 87/2278 [00:12<05:09, 7.08it/s, loss=0.672]
4%|3 | 88/2278 [00:12<05:04, 7.19it/s, loss=0.670]
4%|3 | 89/2278 [00:12<05:09, 7.07it/s, loss=0.676]
4%|3 | 90/2278 [00:12<05:04, 7.19it/s, loss=0.674]
4%|3 | 91/2278 [00:12<05:00, 7.28it/s, loss=0.671]
4%|4 | 92/2278 [00:12<04:57, 7.36it/s, loss=0.671]
4%|4 | 93/2278 [00:12<04:55, 7.40it/s, loss=0.670]
4%|4 | 94/2278 [00:13<04:53, 7.44it/s, loss=0.665]
4%|4 | 95/2278 [00:13<04:54, 7.41it/s, loss=0.674]
4%|4 | 96/2278 [00:13<04:53, 7.43it/s, loss=0.673]
4%|4 | 97/2278 [00:13<05:03, 7.18it/s, loss=0.665]
4%|4 | 98/2278 [00:13<05:02, 7.22it/s, loss=0.665]
4%|4 | 99/2278 [00:13<05:00, 7.25it/s, loss=0.669]
4%|4 | 100/2278 [00:13<04:57, 7.33it/s, loss=0.666]
4%|4 | 101/2278 [00:13<04:56, 7.33it/s, loss=0.666]
4%|4 | 102/2278 [00:14<04:54, 7.38it/s, loss=0.673]
5%|4 | 103/2278 [00:14<04:53, 7.42it/s, loss=0.670]
5%|4 | 104/2278 [00:14<04:50, 7.47it/s, loss=0.675]
5%|4 | 105/2278 [00:14<04:52, 7.43it/s, loss=0.668]
5%|4 | 106/2278 [00:14<05:06, 7.09it/s, loss=0.667]
5%|4 | 107/2278 [00:14<05:02, 7.18it/s, loss=0.671]
5%|4 | 108/2278 [00:14<04:57, 7.29it/s, loss=0.668]
5%|4 | 109/2278 [00:15<04:54, 7.36it/s, loss=0.673]
5%|4 | 110/2278 [00:15<04:53, 7.40it/s, loss=0.673]
5%|4 | 111/2278 [00:15<04:50, 7.45it/s, loss=0.666]
5%|4 | 112/2278 [00:15<04:58, 7.26it/s, loss=0.672]
5%|4 | 113/2278 [00:15<04:55, 7.34it/s, loss=0.666]
5%|5 | 114/2278 [00:15<04:53, 7.37it/s, loss=0.673]
5%|5 | 115/2278 [00:15<04:51, 7.42it/s, loss=0.666]
5%|5 | 116/2278 [00:16<04:50, 7.45it/s, loss=0.670]
5%|5 | 117/2278 [00:16<04:49, 7.46it/s, loss=0.670]
5%|5 | 118/2278 [00:16<04:48, 7.48it/s, loss=0.667]
5%|5 | 119/2278 [00:16<04:48, 7.48it/s, loss=0.674]
5%|5 | 120/2278 [00:16<05:03, 7.11it/s, loss=0.666]
5%|5 | 121/2278 [00:16<05:21, 6.70it/s, loss=0.671]
5%|5 | 122/2278 [00:16<05:27, 6.59it/s, loss=0.672]
5%|5 | 123/2278 [00:17<05:23, 6.66it/s, loss=0.667]
5%|5 | 124/2278 [00:17<05:11, 6.90it/s, loss=0.666]
5%|5 | 125/2278 [00:17<05:03, 7.09it/s, loss=0.668]
6%|5 | 126/2278 [00:17<05:00, 7.16it/s, loss=0.668]
6%|5 | 127/2278 [00:17<05:05, 7.03it/s, loss=0.669]
6%|5 | 128/2278 [00:17<04:59, 7.18it/s, loss=0.674]
6%|5 | 129/2278 [00:17<04:55, 7.28it/s, loss=0.674]
6%|5 | 130/2278 [00:17<04:52, 7.34it/s, loss=0.674]
6%|5 | 131/2278 [00:18<04:49, 7.41it/s, loss=0.671]
6%|5 | 132/2278 [00:18<04:48, 7.44it/s, loss=0.668]
6%|5 | 133/2278 [00:18<04:47, 7.46it/s, loss=0.668]
6%|5 | 134/2278 [00:18<04:46, 7.47it/s, loss=0.667]
6%|5 | 135/2278 [00:18<04:52, 7.33it/s, loss=0.667]
6%|5 | 136/2278 [00:18<04:51, 7.35it/s, loss=0.665]
6%|6 | 137/2278 [00:18<04:49, 7.39it/s, loss=0.670]
6%|6 | 138/2278 [00:19<04:47, 7.44it/s, loss=0.670]
6%|6 | 139/2278 [00:19<04:45, 7.49it/s, loss=0.671]
6%|6 | 140/2278 [00:19<04:51, 7.34it/s, loss=0.666]
6%|6 | 141/2278 [00:19<04:53, 7.28it/s, loss=0.669]
6%|6 | 142/2278 [00:19<05:02, 7.07it/s, loss=0.670]
6%|6 | 143/2278 [00:19<04:56, 7.21it/s, loss=0.667]
6%|6 | 144/2278 [00:19<04:54, 7.23it/s, loss=0.671]
6%|6 | 145/2278 [00:20<04:50, 7.33it/s, loss=0.665]
6%|6 | 146/2278 [00:20<04:48, 7.38it/s, loss=0.669]
6%|6 | 147/2278 [00:20<04:47, 7.40it/s, loss=0.663]
6%|6 | 148/2278 [00:20<04:46, 7.43it/s, loss=0.672]
7%|6 | 149/2278 [00:20<04:45, 7.45it/s, loss=0.671]
7%|6 | 150/2278 [00:20<04:57, 7.16it/s, loss=0.668]
7%|6 | 151/2278 [00:20<04:51, 7.29it/s, loss=0.668]
7%|6 | 152/2278 [00:20<04:48, 7.37it/s, loss=0.670]
7%|6 | 153/2278 [00:21<04:48, 7.37it/s, loss=0.662]
7%|6 | 154/2278 [00:21<04:46, 7.42it/s, loss=0.668]
7%|6 | 155/2278 [00:21<04:45, 7.44it/s, loss=0.671]
7%|6 | 156/2278 [00:21<04:47, 7.37it/s, loss=0.667]
7%|6 | 157/2278 [00:21<04:46, 7.42it/s, loss=0.670]
7%|6 | 158/2278 [00:21<04:48, 7.36it/s, loss=0.673]
7%|6 | 159/2278 [00:21<04:45, 7.43it/s, loss=0.678]
7%|7 | 160/2278 [00:22<04:43, 7.48it/s, loss=0.666]
7%|7 | 161/2278 [00:22<04:42, 7.48it/s, loss=0.675]
7%|7 | 162/2278 [00:22<04:41, 7.53it/s, loss=0.666]
7%|7 | 163/2278 [00:22<04:45, 7.41it/s, loss=0.664]
7%|7 | 164/2278 [00:22<04:45, 7.41it/s, loss=0.668]
7%|7 | 165/2278 [00:22<04:53, 7.19it/s, loss=0.665]
7%|7 | 166/2278 [00:22<05:09, 6.82it/s, loss=0.671]
7%|7 | 167/2278 [00:23<05:12, 6.75it/s, loss=0.666]
7%|7 | 168/2278 [00:23<05:24, 6.51it/s, loss=0.665]
7%|7 | 169/2278 [00:23<05:10, 6.79it/s, loss=0.668]
7%|7 | 170/2278 [00:23<05:00, 7.00it/s, loss=0.664]
8%|7 | 171/2278 [00:23<04:54, 7.14it/s, loss=0.667]
8%|7 | 172/2278 [00:23<04:57, 7.08it/s, loss=0.670]
8%|7 | 173/2278 [00:23<05:35, 6.27it/s, loss=0.667]
8%|7 | 174/2278 [00:24<05:18, 6.61it/s, loss=0.673]
8%|7 | 175/2278 [00:24<05:06, 6.86it/s, loss=0.667]
8%|7 | 176/2278 [00:24<04:59, 7.02it/s, loss=0.666]
8%|7 | 177/2278 [00:24<04:54, 7.14it/s, loss=0.668]
8%|7 | 178/2278 [00:24<04:50, 7.23it/s, loss=0.669]
8%|7 | 179/2278 [00:24<04:48, 7.29it/s, loss=0.671]
8%|7 | 180/2278 [00:24<04:49, 7.25it/s, loss=0.664]
8%|7 | 181/2278 [00:25<04:48, 7.27it/s, loss=0.668]
8%|7 | 182/2278 [00:25<04:45, 7.34it/s, loss=0.668]
8%|8 | 183/2278 [00:25<04:44, 7.35it/s, loss=0.666]
8%|8 | 184/2278 [00:25<04:44, 7.35it/s, loss=0.668]
8%|8 | 185/2278 [00:25<04:43, 7.39it/s, loss=0.671]
8%|8 | 186/2278 [00:25<04:41, 7.44it/s, loss=0.665]
8%|8 | 187/2278 [00:25<04:49, 7.21it/s, loss=0.668]
8%|8 | 188/2278 [00:25<04:46, 7.30it/s, loss=0.669]
8%|8 | 189/2278 [00:26<04:43, 7.38it/s, loss=0.674]
8%|8 | 190/2278 [00:26<04:41, 7.41it/s, loss=0.666]
8%|8 | 191/2278 [00:26<04:41, 7.41it/s, loss=0.662]
8%|8 | 192/2278 [00:26<04:42, 7.40it/s, loss=0.669]
8%|8 | 193/2278 [00:26<04:40, 7.43it/s, loss=0.665]
9%|8 | 194/2278 [00:26<04:43, 7.35it/s, loss=0.669]
9%|8 | 195/2278 [00:26<04:52, 7.12it/s, loss=0.664]
9%|8 | 196/2278 [00:27<05:00, 6.92it/s, loss=0.670]
9%|8 | 197/2278 [00:27<05:15, 6.59it/s, loss=0.671]
9%|8 | 198/2278 [00:27<05:07, 6.77it/s, loss=0.669]
9%|8 | 199/2278 [00:27<04:58, 6.96it/s, loss=0.669]
9%|8 | 200/2278 [00:27<04:53, 7.07it/s, loss=0.667]
9%|8 | 201/2278 [00:27<04:49, 7.18it/s, loss=0.668]
9%|8 | 202/2278 [00:27<04:52, 7.09it/s, loss=0.668]
9%|8 | 203/2278 [00:28<04:47, 7.21it/s, loss=0.665]
9%|8 | 204/2278 [00:28<04:45, 7.28it/s, loss=0.670]
9%|8 | 205/2278 [00:28<04:44, 7.28it/s, loss=0.668]
9%|9 | 206/2278 [00:28<04:42, 7.33it/s, loss=0.667]
9%|9 | 207/2278 [00:28<04:43, 7.30it/s, loss=0.666]
9%|9 | 208/2278 [00:28<04:52, 7.08it/s, loss=0.664]
9%|9 | 209/2278 [00:28<05:00, 6.88it/s, loss=0.671]
9%|9 | 210/2278 [00:29<04:59, 6.90it/s, loss=0.673]
9%|9 | 211/2278 [00:29<04:53, 7.05it/s, loss=0.662]
9%|9 | 212/2278 [00:29<04:47, 7.19it/s, loss=0.667]
9%|9 | 213/2278 [00:29<04:42, 7.31it/s, loss=0.668]
9%|9 | 214/2278 [00:29<04:39, 7.40it/s, loss=0.665]
9%|9 | 215/2278 [00:29<04:38, 7.40it/s, loss=0.665]
9%|9 | 216/2278 [00:29<04:38, 7.41it/s, loss=0.670]
10%|9 | 217/2278 [00:30<04:44, 7.24it/s, loss=0.666]
10%|9 | 218/2278 [00:30<04:41, 7.32it/s, loss=0.667]
10%|9 | 219/2278 [00:30<04:41, 7.31it/s, loss=0.661]
10%|9 | 220/2278 [00:30<04:38, 7.39it/s, loss=0.667]
10%|9 | 221/2278 [00:30<04:37, 7.42it/s, loss=0.666]
10%|9 | 222/2278 [00:30<04:36, 7.45it/s, loss=0.665]
10%|9 | 223/2278 [00:30<04:35, 7.47it/s, loss=0.666]
10%|9 | 224/2278 [00:30<04:34, 7.48it/s, loss=0.668]
10%|9 | 225/2278 [00:31<04:41, 7.31it/s, loss=0.662]
10%|9 | 226/2278 [00:31<04:39, 7.34it/s, loss=0.665]
10%|9 | 227/2278 [00:31<04:37, 7.38it/s, loss=0.662]
10%|# | 228/2278 [00:31<04:37, 7.38it/s, loss=0.667]
10%|# | 229/2278 [00:31<04:35, 7.43it/s, loss=0.668]
10%|# | 230/2278 [00:31<04:34, 7.46it/s, loss=0.666]
10%|# | 231/2278 [00:31<04:34, 7.47it/s, loss=0.666]
10%|# | 232/2278 [00:32<04:33, 7.47it/s, loss=0.672]
10%|# | 233/2278 [00:32<04:37, 7.36it/s, loss=0.674]
10%|# | 234/2278 [00:32<04:36, 7.40it/s, loss=0.667]
10%|# | 235/2278 [00:32<04:34, 7.44it/s, loss=0.667]
10%|# | 236/2278 [00:32<04:33, 7.48it/s, loss=0.666]
10%|# | 237/2278 [00:32<04:32, 7.49it/s, loss=0.667]
10%|# | 238/2278 [00:32<04:31, 7.50it/s, loss=0.669]
10%|# | 239/2278 [00:32<04:31, 7.51it/s, loss=0.666]
11%|# | 240/2278 [00:33<04:40, 7.27it/s, loss=0.665]
11%|# | 241/2278 [00:33<04:38, 7.33it/s, loss=0.662]
11%|# | 242/2278 [00:33<04:35, 7.39it/s, loss=0.657]
11%|# | 243/2278 [00:33<04:33, 7.44it/s, loss=0.668]
11%|# | 244/2278 [00:33<04:32, 7.46it/s, loss=0.667]
11%|# | 245/2278 [00:33<04:31, 7.50it/s, loss=0.670]
11%|# | 246/2278 [00:33<04:31, 7.47it/s, loss=0.668]
11%|# | 247/2278 [00:34<04:32, 7.44it/s, loss=0.666]
11%|# | 248/2278 [00:34<04:38, 7.29it/s, loss=0.665]
11%|# | 249/2278 [00:34<04:35, 7.36it/s, loss=0.667]
11%|# | 250/2278 [00:34<04:34, 7.38it/s, loss=0.665]
11%|#1 | 251/2278 [00:34<04:32, 7.43it/s, loss=0.667]
11%|#1 | 252/2278 [00:34<04:32, 7.44it/s, loss=0.669]
11%|#1 | 253/2278 [00:34<04:30, 7.48it/s, loss=0.665]
11%|#1 | 254/2278 [00:35<04:30, 7.47it/s, loss=0.668]
11%|#1 | 255/2278 [00:35<04:29, 7.50it/s, loss=0.665]
11%|#1 | 256/2278 [00:35<04:39, 7.23it/s, loss=0.668]
11%|#1 | 257/2278 [00:35<04:36, 7.30it/s, loss=0.666]
11%|#1 | 258/2278 [00:35<04:33, 7.37it/s, loss=0.663]
11%|#1 | 259/2278 [00:35<04:33, 7.39it/s, loss=0.662]
11%|#1 | 260/2278 [00:35<04:32, 7.41it/s, loss=0.667]
11%|#1 | 261/2278 [00:35<04:31, 7.43it/s, loss=0.669]
12%|#1 | 262/2278 [00:36<04:32, 7.39it/s, loss=0.664]
12%|#1 | 263/2278 [00:36<04:43, 7.11it/s, loss=0.664]
12%|#1 | 264/2278 [00:36<04:58, 6.75it/s, loss=0.665]
12%|#1 | 265/2278 [00:36<05:05, 6.58it/s, loss=0.672]
12%|#1 | 266/2278 [00:36<05:00, 6.69it/s, loss=0.660]
12%|#1 | 267/2278 [00:36<04:58, 6.73it/s, loss=0.667]
12%|#1 | 268/2278 [00:37<04:55, 6.80it/s, loss=0.665]
12%|#1 | 269/2278 [00:37<04:47, 6.99it/s, loss=0.669]
12%|#1 | 270/2278 [00:37<04:49, 6.93it/s, loss=0.665]
12%|#1 | 271/2278 [00:37<05:02, 6.63it/s, loss=0.663]
12%|#1 | 272/2278 [00:37<05:04, 6.58it/s, loss=0.666]
12%|#1 | 273/2278 [00:37<04:59, 6.69it/s, loss=0.663]
12%|#2 | 274/2278 [00:37<04:49, 6.92it/s, loss=0.664]
12%|#2 | 275/2278 [00:38<04:46, 6.99it/s, loss=0.665]
12%|#2 | 276/2278 [00:38<04:39, 7.16it/s, loss=0.665]
12%|#2 | 277/2278 [00:38<04:55, 6.76it/s, loss=0.666]
12%|#2 | 278/2278 [00:38<05:06, 6.52it/s, loss=0.663]
12%|#2 | 279/2278 [00:38<05:01, 6.64it/s, loss=0.665]
12%|#2 | 280/2278 [00:38<05:10, 6.44it/s, loss=0.666]
12%|#2 | 281/2278 [00:38<04:57, 6.72it/s, loss=0.663]
12%|#2 | 282/2278 [00:39<04:46, 6.96it/s, loss=0.666]
12%|#2 | 283/2278 [00:39<04:45, 6.98it/s, loss=0.660]
12%|#2 | 284/2278 [00:39<04:49, 6.88it/s, loss=0.660]
13%|#2 | 285/2278 [00:39<04:41, 7.08it/s, loss=0.663]
13%|#2 | 286/2278 [00:39<04:41, 7.07it/s, loss=0.666]
13%|#2 | 287/2278 [00:39<04:37, 7.18it/s, loss=0.664]
13%|#2 | 288/2278 [00:39<04:52, 6.79it/s, loss=0.659]
13%|#2 | 289/2278 [00:40<04:43, 7.01it/s, loss=0.664]
13%|#2 | 290/2278 [00:40<04:36, 7.20it/s, loss=0.662]
13%|#2 | 291/2278 [00:40<04:31, 7.33it/s, loss=0.667]
13%|#2 | 292/2278 [00:40<04:35, 7.20it/s, loss=0.665]
13%|#2 | 293/2278 [00:40<04:39, 7.09it/s, loss=0.672]
13%|#2 | 294/2278 [00:40<04:34, 7.22it/s, loss=0.662]
13%|#2 | 295/2278 [00:40<04:32, 7.29it/s, loss=0.663]
13%|#2 | 296/2278 [00:41<04:29, 7.34it/s, loss=0.666]
13%|#3 | 297/2278 [00:41<04:27, 7.40it/s, loss=0.662]
13%|#3 | 298/2278 [00:41<04:32, 7.27it/s, loss=0.671]
13%|#3 | 299/2278 [00:41<04:38, 7.09it/s, loss=0.663]
13%|#3 | 300/2278 [00:41<04:34, 7.22it/s, loss=0.665]
13%|#3 | 301/2278 [00:41<04:31, 7.27it/s, loss=0.666]
13%|#3 | 302/2278 [00:41<04:29, 7.34it/s, loss=0.663]
13%|#3 | 303/2278 [00:41<04:27, 7.39it/s, loss=0.665]
13%|#3 | 304/2278 [00:42<04:26, 7.40it/s, loss=0.658]
13%|#3 | 305/2278 [00:42<04:25, 7.44it/s, loss=0.660]
13%|#3 | 306/2278 [00:42<04:23, 7.48it/s, loss=0.663]
13%|#3 | 307/2278 [00:42<04:28, 7.34it/s, loss=0.666]
14%|#3 | 308/2278 [00:42<04:32, 7.22it/s, loss=0.667]
14%|#3 | 309/2278 [00:42<04:35, 7.14it/s, loss=0.666]
14%|#3 | 310/2278 [00:42<04:33, 7.19it/s, loss=0.665]
14%|#3 | 311/2278 [00:43<04:51, 6.75it/s, loss=0.664]
14%|#3 | 312/2278 [00:43<04:50, 6.76it/s, loss=0.668]
14%|#3 | 313/2278 [00:43<04:40, 7.01it/s, loss=0.663]
14%|#3 | 314/2278 [00:43<04:42, 6.95it/s, loss=0.663]
14%|#3 | 315/2278 [00:43<04:54, 6.66it/s, loss=0.661]
14%|#3 | 316/2278 [00:43<05:07, 6.38it/s, loss=0.665]
14%|#3 | 317/2278 [00:44<05:00, 6.53it/s, loss=0.663]
14%|#3 | 318/2278 [00:44<05:02, 6.48it/s, loss=0.663]
14%|#4 | 319/2278 [00:44<04:58, 6.56it/s, loss=0.660]
14%|#4 | 320/2278 [00:44<04:56, 6.61it/s, loss=0.664]
14%|#4 | 321/2278 [00:44<05:05, 6.41it/s, loss=0.665]
14%|#4 | 322/2278 [00:44<04:54, 6.64it/s, loss=0.662]
14%|#4 | 323/2278 [00:44<04:44, 6.87it/s, loss=0.663]
14%|#4 | 324/2278 [00:45<04:36, 7.06it/s, loss=0.659]
14%|#4 | 325/2278 [00:45<04:31, 7.20it/s, loss=0.662]
14%|#4 | 326/2278 [00:45<04:27, 7.30it/s, loss=0.663]
14%|#4 | 327/2278 [00:45<04:24, 7.36it/s, loss=0.661]
14%|#4 | 328/2278 [00:45<04:25, 7.35it/s, loss=0.665]
14%|#4 | 329/2278 [00:45<04:43, 6.88it/s, loss=0.659]
14%|#4 | 330/2278 [00:45<04:43, 6.88it/s, loss=0.664]
15%|#4 | 331/2278 [00:46<04:58, 6.53it/s, loss=0.667]
15%|#4 | 332/2278 [00:46<04:45, 6.81it/s, loss=0.660]
15%|#4 | 333/2278 [00:46<04:37, 7.00it/s, loss=0.664]
15%|#4 | 334/2278 [00:46<04:31, 7.16it/s, loss=0.662]
15%|#4 | 335/2278 [00:46<04:27, 7.26it/s, loss=0.657]
15%|#4 | 336/2278 [00:46<04:26, 7.29it/s, loss=0.668]
15%|#4 | 337/2278 [00:46<04:23, 7.35it/s, loss=0.664]
15%|#4 | 338/2278 [00:47<04:22, 7.38it/s, loss=0.660]
15%|#4 | 339/2278 [00:47<04:21, 7.40it/s, loss=0.662]
15%|#4 | 340/2278 [00:47<04:23, 7.35it/s, loss=0.672]
15%|#4 | 341/2278 [00:47<04:21, 7.40it/s, loss=0.662]
15%|#5 | 342/2278 [00:47<04:19, 7.45it/s, loss=0.664]
15%|#5 | 343/2278 [00:47<04:27, 7.22it/s, loss=0.661]
15%|#5 | 344/2278 [00:47<04:23, 7.33it/s, loss=0.661]
15%|#5 | 345/2278 [00:47<04:21, 7.38it/s, loss=0.662]
15%|#5 | 346/2278 [00:48<04:19, 7.43it/s, loss=0.661]
15%|#5 | 347/2278 [00:48<04:18, 7.47it/s, loss=0.659]
15%|#5 | 348/2278 [00:48<04:17, 7.50it/s, loss=0.663]
15%|#5 | 349/2278 [00:48<04:16, 7.51it/s, loss=0.660]
15%|#5 | 350/2278 [00:48<04:18, 7.46it/s, loss=0.660]
15%|#5 | 351/2278 [00:48<04:31, 7.11it/s, loss=0.668]
15%|#5 | 352/2278 [00:48<04:25, 7.25it/s, loss=0.663]
15%|#5 | 353/2278 [00:49<04:22, 7.33it/s, loss=0.661]
16%|#5 | 354/2278 [00:49<04:21, 7.37it/s, loss=0.660]
16%|#5 | 355/2278 [00:49<04:19, 7.40it/s, loss=0.663]
16%|#5 | 356/2278 [00:49<04:18, 7.44it/s, loss=0.665]
16%|#5 | 357/2278 [00:49<04:18, 7.44it/s, loss=0.658]
16%|#5 | 358/2278 [00:49<04:18, 7.42it/s, loss=0.663]
16%|#5 | 359/2278 [00:49<04:21, 7.33it/s, loss=0.661]
16%|#5 | 360/2278 [00:49<04:19, 7.38it/s, loss=0.662]
16%|#5 | 361/2278 [00:50<04:20, 7.37it/s, loss=0.664]
16%|#5 | 362/2278 [00:50<04:20, 7.36it/s, loss=0.661]
16%|#5 | 363/2278 [00:50<04:18, 7.41it/s, loss=0.661]
16%|#5 | 364/2278 [00:50<04:17, 7.42it/s, loss=0.661]
16%|#6 | 365/2278 [00:50<04:16, 7.45it/s, loss=0.657]
16%|#6 | 366/2278 [00:50<04:24, 7.23it/s, loss=0.662]
16%|#6 | 367/2278 [00:50<04:20, 7.32it/s, loss=0.662]
16%|#6 | 368/2278 [00:51<04:19, 7.37it/s, loss=0.664]
16%|#6 | 369/2278 [00:51<04:19, 7.35it/s, loss=0.670]
16%|#6 | 370/2278 [00:51<04:18, 7.39it/s, loss=0.664]
16%|#6 | 371/2278 [00:51<04:18, 7.37it/s, loss=0.660]
16%|#6 | 372/2278 [00:51<04:17, 7.39it/s, loss=0.655]
16%|#6 | 373/2278 [00:51<04:16, 7.43it/s, loss=0.663]
16%|#6 | 374/2278 [00:51<04:24, 7.19it/s, loss=0.663]
16%|#6 | 375/2278 [00:52<04:21, 7.29it/s, loss=0.660]
17%|#6 | 376/2278 [00:52<04:17, 7.37it/s, loss=0.662]
17%|#6 | 377/2278 [00:52<04:18, 7.35it/s, loss=0.662]
17%|#6 | 378/2278 [00:52<04:17, 7.39it/s, loss=0.665]
17%|#6 | 379/2278 [00:52<04:18, 7.36it/s, loss=0.660]
17%|#6 | 380/2278 [00:52<04:17, 7.36it/s, loss=0.659]
17%|#6 | 381/2278 [00:52<04:17, 7.37it/s, loss=0.661]
17%|#6 | 382/2278 [00:52<04:18, 7.32it/s, loss=0.662]
17%|#6 | 383/2278 [00:53<04:18, 7.34it/s, loss=0.656]
17%|#6 | 384/2278 [00:53<04:20, 7.26it/s, loss=0.665]
17%|#6 | 385/2278 [00:53<04:33, 6.93it/s, loss=0.663]
17%|#6 | 386/2278 [00:53<04:26, 7.10it/s, loss=0.660]
17%|#6 | 387/2278 [00:53<04:22, 7.22it/s, loss=0.660]
17%|#7 | 388/2278 [00:53<04:20, 7.26it/s, loss=0.662]
17%|#7 | 389/2278 [00:53<04:28, 7.02it/s, loss=0.663]
17%|#7 | 390/2278 [00:54<04:36, 6.84it/s, loss=0.669]
17%|#7 | 391/2278 [00:54<04:54, 6.41it/s, loss=0.658]
17%|#7 | 392/2278 [00:54<05:00, 6.27it/s, loss=0.657]
17%|#7 | 393/2278 [00:54<04:56, 6.36it/s, loss=0.667]
17%|#7 | 394/2278 [00:54<04:47, 6.54it/s, loss=0.662]
17%|#7 | 395/2278 [00:54<04:37, 6.79it/s, loss=0.660]
17%|#7 | 396/2278 [00:55<04:34, 6.85it/s, loss=0.657]
17%|#7 | 397/2278 [00:55<04:27, 7.03it/s, loss=0.663]
17%|#7 | 398/2278 [00:55<04:21, 7.19it/s, loss=0.655]
18%|#7 | 399/2278 [00:55<04:17, 7.28it/s, loss=0.661]
18%|#7 | 400/2278 [00:55<04:15, 7.34it/s, loss=0.663]
18%|#7 | 401/2278 [00:55<04:31, 6.90it/s, loss=0.659]
18%|#7 | 402/2278 [00:55<04:33, 6.86it/s, loss=0.662]
18%|#7 | 403/2278 [00:56<04:36, 6.78it/s, loss=0.659]
18%|#7 | 404/2278 [00:56<04:27, 7.00it/s, loss=0.658]
18%|#7 | 405/2278 [00:56<04:21, 7.15it/s, loss=0.653]
18%|#7 | 406/2278 [00:56<04:17, 7.26it/s, loss=0.660]
18%|#7 | 407/2278 [00:56<04:18, 7.25it/s, loss=0.666]
18%|#7 | 408/2278 [00:56<04:17, 7.25it/s, loss=0.663]
18%|#7 | 409/2278 [00:56<04:16, 7.30it/s, loss=0.658]
18%|#7 | 410/2278 [00:56<04:15, 7.31it/s, loss=0.661]
18%|#8 | 411/2278 [00:57<04:22, 7.11it/s, loss=0.657]
18%|#8 | 412/2278 [00:57<04:18, 7.23it/s, loss=0.656]
18%|#8 | 413/2278 [00:57<04:16, 7.26it/s, loss=0.659]
18%|#8 | 414/2278 [00:57<04:16, 7.27it/s, loss=0.661]
18%|#8 | 415/2278 [00:57<04:14, 7.32it/s, loss=0.665]
18%|#8 | 416/2278 [00:57<04:13, 7.35it/s, loss=0.666]
18%|#8 | 417/2278 [00:57<04:12, 7.36it/s, loss=0.663]
18%|#8 | 418/2278 [00:58<04:20, 7.13it/s, loss=0.662]
18%|#8 | 419/2278 [00:58<04:41, 6.61it/s, loss=0.660]
18%|#8 | 420/2278 [00:58<04:38, 6.66it/s, loss=0.664]
18%|#8 | 421/2278 [00:58<04:29, 6.88it/s, loss=0.659]
19%|#8 | 422/2278 [00:58<04:23, 7.05it/s, loss=0.665]
19%|#8 | 423/2278 [00:58<04:17, 7.21it/s, loss=0.660]
19%|#8 | 424/2278 [00:58<04:13, 7.30it/s, loss=0.657]
19%|#8 | 425/2278 [00:59<04:19, 7.15it/s, loss=0.664]
19%|#8 | 426/2278 [00:59<04:30, 6.86it/s, loss=0.660]
19%|#8 | 427/2278 [00:59<04:23, 7.04it/s, loss=0.658]
19%|#8 | 428/2278 [00:59<04:17, 7.19it/s, loss=0.661]
19%|#8 | 429/2278 [00:59<04:14, 7.27it/s, loss=0.657]
19%|#8 | 430/2278 [00:59<04:11, 7.35it/s, loss=0.663]
19%|#8 | 431/2278 [00:59<04:08, 7.42it/s, loss=0.661]
19%|#8 | 432/2278 [01:00<04:07, 7.45it/s, loss=0.660]
19%|#9 | 433/2278 [01:00<04:13, 7.28it/s, loss=0.660]
19%|#9 | 434/2278 [01:00<04:10, 7.36it/s, loss=0.659]
19%|#9 | 435/2278 [01:00<04:10, 7.37it/s, loss=0.661]
19%|#9 | 436/2278 [01:00<04:08, 7.42it/s, loss=0.656]
19%|#9 | 437/2278 [01:00<04:07, 7.44it/s, loss=0.662]
19%|#9 | 438/2278 [01:00<04:07, 7.44it/s, loss=0.664]
19%|#9 | 439/2278 [01:01<04:06, 7.45it/s, loss=0.659]
19%|#9 | 440/2278 [01:01<04:07, 7.43it/s, loss=0.664]
19%|#9 | 441/2278 [01:01<04:07, 7.43it/s, loss=0.664]
19%|#9 | 442/2278 [01:01<04:06, 7.44it/s, loss=0.655]
19%|#9 | 443/2278 [01:01<04:06, 7.45it/s, loss=0.656]
19%|#9 | 444/2278 [01:01<04:05, 7.47it/s, loss=0.660]
20%|#9 | 445/2278 [01:01<04:05, 7.48it/s, loss=0.661]
20%|#9 | 446/2278 [01:01<04:05, 7.47it/s, loss=0.659]
20%|#9 | 447/2278 [01:02<04:04, 7.48it/s, loss=0.666]
20%|#9 | 448/2278 [01:02<04:12, 7.25it/s, loss=0.661]
20%|#9 | 449/2278 [01:02<04:09, 7.34it/s, loss=0.655]
20%|#9 | 450/2278 [01:02<04:06, 7.42it/s, loss=0.662]
20%|#9 | 451/2278 [01:02<04:05, 7.44it/s, loss=0.660]
20%|#9 | 452/2278 [01:02<04:04, 7.46it/s, loss=0.660]
20%|#9 | 453/2278 [01:02<04:04, 7.47it/s, loss=0.661]
20%|#9 | 454/2278 [01:03<04:05, 7.43it/s, loss=0.661]
20%|#9 | 455/2278 [01:03<04:04, 7.45it/s, loss=0.659]
20%|## | 456/2278 [01:03<04:12, 7.21it/s, loss=0.655]
20%|## | 457/2278 [01:03<04:09, 7.31it/s, loss=0.664]
20%|## | 458/2278 [01:03<04:07, 7.37it/s, loss=0.660]
20%|## | 459/2278 [01:03<04:05, 7.42it/s, loss=0.659]
20%|## | 460/2278 [01:03<04:03, 7.47it/s, loss=0.661]
20%|## | 461/2278 [01:03<04:02, 7.50it/s, loss=0.659]
20%|## | 462/2278 [01:04<04:01, 7.51it/s, loss=0.657]
20%|## | 463/2278 [01:04<04:01, 7.53it/s, loss=0.661]
20%|## | 464/2278 [01:04<04:07, 7.32it/s, loss=0.661]
20%|## | 465/2278 [01:04<04:05, 7.39it/s, loss=0.656]
20%|## | 466/2278 [01:04<04:03, 7.43it/s, loss=0.663]
21%|## | 467/2278 [01:04<04:02, 7.46it/s, loss=0.662]
21%|## | 468/2278 [01:04<04:01, 7.49it/s, loss=0.657]
21%|## | 469/2278 [01:05<04:00, 7.53it/s, loss=0.656]
21%|## | 470/2278 [01:05<04:03, 7.41it/s, loss=0.662]
21%|## | 471/2278 [01:05<04:06, 7.32it/s, loss=0.660]
21%|## | 472/2278 [01:05<04:23, 6.85it/s, loss=0.664]
21%|## | 473/2278 [01:05<04:33, 6.59it/s, loss=0.668]
21%|## | 474/2278 [01:05<04:41, 6.41it/s, loss=0.664]
21%|## | 475/2278 [01:05<04:43, 6.36it/s, loss=0.659]
21%|## | 476/2278 [01:06<04:30, 6.66it/s, loss=0.657]
21%|## | 477/2278 [01:06<04:21, 6.89it/s, loss=0.662]
21%|## | 478/2278 [01:06<04:21, 6.87it/s, loss=0.658]
21%|##1 | 479/2278 [01:06<04:15, 7.05it/s, loss=0.657]
21%|##1 | 480/2278 [01:06<04:10, 7.18it/s, loss=0.652]
21%|##1 | 481/2278 [01:06<04:08, 7.22it/s, loss=0.660]
21%|##1 | 482/2278 [01:06<04:05, 7.30it/s, loss=0.660]
21%|##1 | 483/2278 [01:07<04:04, 7.35it/s, loss=0.655]
21%|##1 | 484/2278 [01:07<04:02, 7.39it/s, loss=0.665]
21%|##1 | 485/2278 [01:07<04:03, 7.38it/s, loss=0.658]
21%|##1 | 486/2278 [01:07<04:10, 7.17it/s, loss=0.652]
21%|##1 | 487/2278 [01:07<04:06, 7.26it/s, loss=0.656]
21%|##1 | 488/2278 [01:07<04:05, 7.30it/s, loss=0.659]
21%|##1 | 489/2278 [01:07<04:03, 7.36it/s, loss=0.659]
22%|##1 | 490/2278 [01:08<04:01, 7.40it/s, loss=0.664]
22%|##1 | 491/2278 [01:08<04:00, 7.43it/s, loss=0.651]
22%|##1 | 492/2278 [01:08<03:59, 7.44it/s, loss=0.660]
22%|##1 | 493/2278 [01:08<03:59, 7.47it/s, loss=0.659]
22%|##1 | 494/2278 [01:08<04:00, 7.43it/s, loss=0.658]
22%|##1 | 495/2278 [01:08<03:58, 7.46it/s, loss=0.658]
22%|##1 | 496/2278 [01:08<03:58, 7.49it/s, loss=0.662]
22%|##1 | 497/2278 [01:08<03:56, 7.52it/s, loss=0.656]
22%|##1 | 498/2278 [01:09<03:59, 7.44it/s, loss=0.652]
22%|##1 | 499/2278 [01:09<03:58, 7.45it/s, loss=0.660]Converges at iteration 19
Epoch 0 Validation Accuracy 0.5975703882680626 Test Accuracy 0.5877003477151617
Evaluating Performance with Link Prediction (Optional)¶
In practice, it is more common to evaluate the link prediction model to see whether it can predict new edges. There are different evaluation metrics such as AUC or various metrics from information retrieval. Ultimately, they require the model to predict one scalar score given a node pair among a set of node pairs.
dgl.dataloading.EdgeDataLoader allows you to iterate over
the edges of a new graph with the same nodes, while performing
neighbor sampling on the original graph with g_sampling argument.
This functionality enables convenient evaluation of a link prediction
model.
Assuming that you have the following test set with labels, where
test_pos_src and test_pos_dst are ground truth node pairs
with edges in between (or positive pairs), and test_neg_src
and test_neg_dst are ground truth node pairs without edges
in between (or negative pairs).
# Positive pairs
test_pos_src, test_pos_dst = graph.edges()
# Negative pairs
test_neg_src = test_pos_src
test_neg_dst = torch.randint(0, graph.num_nodes(), (graph.num_edges(),))
First you need to construct a graph for dgl.dataloading.EdgeDataLoader
to iterate on, i.e. with the testing node pairs as edges.
You also need to label the edges, 1 if positive and 0 if negative.
test_src = torch.cat([test_pos_src, test_pos_dst])
test_dst = torch.cat([test_neg_src, test_neg_dst])
test_graph = dgl.graph((test_src, test_dst), num_nodes=graph.num_nodes())
test_graph.edata['label'] = torch.cat(
[torch.ones_like(test_pos_src), torch.zeros_like(test_neg_src)])
Then you could create a new EdgeDataLoader instance that
iterates on the new test_graph, but uses the original graph
for neighbor sampling.
Note that you do not need negative sampling in this dataloader: the negative pairs are already in the new test graph.
test_dataloader = dgl.dataloading.EdgeDataLoader(
# The following arguments are specific to EdgeDataLoader.
test_graph, # The graph to iterate edges over
torch.arange(test_graph.number_of_edges()), # The edges to iterate over
sampler, # The neighbor sampler
device=device, # Put the MFGs on CPU or GPU
g_sampling=graph, # Graph to sample neighbors
# The following arguments are inherited from PyTorch DataLoader.
batch_size=1024, # Batch size
shuffle=True, # Whether to shuffle the nodes for every epoch
drop_last=False, # Whether to drop the last incomplete batch
num_workers=0 # Number of sampler processes
)
The rest is similar to training except that you no longer compute the gradients, and you collect all the scores and ground truth labels for final metric calculation.
Note
If the graph does not change, you can also precompute all the
node representations beforehand with inference function.
You can then feed the precomputed results directly into the
predictor without passing the MFGs into the model.
test_preds = []
test_labels = []
with tqdm.tqdm(test_dataloader) as tq, torch.no_grad():
for step, (input_nodes, pair_graph, mfgs) in enumerate(tq):
# feature copy from CPU to GPU takes place here
inputs = mfgs[0].srcdata['feat']
outputs = model(mfgs, inputs)
test_preds.append(predictor(pair_graph, outputs))
test_labels.append(pair_graph.edata['label'])
test_preds = torch.cat(test_preds).cpu().numpy()
test_labels = torch.cat(test_labels).cpu().numpy()
auc = sklearn.metrics.roc_auc_score(test_labels, test_preds)
print('Link Prediction AUC:', auc)
Out:
0%| | 0/4556 [00:00<?, ?it/s]
0%| | 1/4556 [00:00<26:05, 2.91it/s]
0%| | 5/4556 [00:00<18:53, 4.02it/s]
0%| | 9/4556 [00:00<13:51, 5.47it/s]
0%| | 12/4556 [00:00<10:32, 7.18it/s]
0%| | 16/4556 [00:00<08:00, 9.44it/s]
0%| | 20/4556 [00:00<06:14, 12.11it/s]
1%| | 24/4556 [00:01<05:00, 15.08it/s]
1%| | 28/4556 [00:01<04:08, 18.24it/s]
1%| | 32/4556 [00:01<03:31, 21.36it/s]
1%| | 36/4556 [00:01<03:08, 23.99it/s]
1%| | 40/4556 [00:01<02:49, 26.57it/s]
1%| | 44/4556 [00:01<02:37, 28.72it/s]
1%|1 | 48/4556 [00:01<02:28, 30.41it/s]
1%|1 | 52/4556 [00:01<02:21, 31.77it/s]
1%|1 | 56/4556 [00:01<02:17, 32.84it/s]
1%|1 | 60/4556 [00:02<02:13, 33.66it/s]
1%|1 | 64/4556 [00:02<02:11, 34.20it/s]
1%|1 | 68/4556 [00:02<02:09, 34.57it/s]
2%|1 | 72/4556 [00:02<02:10, 34.32it/s]
2%|1 | 76/4556 [00:02<02:09, 34.64it/s]
2%|1 | 80/4556 [00:02<02:08, 34.80it/s]
2%|1 | 84/4556 [00:02<02:07, 34.98it/s]
2%|1 | 88/4556 [00:02<02:07, 35.03it/s]
2%|2 | 92/4556 [00:02<02:07, 35.12it/s]
2%|2 | 96/4556 [00:03<02:06, 35.15it/s]
2%|2 | 100/4556 [00:03<02:06, 35.21it/s]
2%|2 | 104/4556 [00:03<02:06, 35.28it/s]
2%|2 | 108/4556 [00:03<02:07, 34.76it/s]
2%|2 | 112/4556 [00:03<02:07, 34.93it/s]
3%|2 | 116/4556 [00:03<02:06, 35.09it/s]
3%|2 | 120/4556 [00:03<02:06, 35.07it/s]
3%|2 | 124/4556 [00:03<02:06, 35.05it/s]
3%|2 | 128/4556 [00:03<02:06, 35.06it/s]
3%|2 | 132/4556 [00:04<02:05, 35.12it/s]
3%|2 | 136/4556 [00:04<02:05, 35.17it/s]
3%|3 | 140/4556 [00:04<02:05, 35.23it/s]
3%|3 | 144/4556 [00:04<02:06, 34.79it/s]
3%|3 | 148/4556 [00:04<02:06, 34.80it/s]
3%|3 | 152/4556 [00:04<02:06, 34.84it/s]
3%|3 | 156/4556 [00:04<02:06, 34.88it/s]
4%|3 | 160/4556 [00:04<02:06, 34.70it/s]
4%|3 | 164/4556 [00:05<02:06, 34.81it/s]
4%|3 | 168/4556 [00:05<02:05, 34.89it/s]
4%|3 | 172/4556 [00:05<02:05, 34.92it/s]
4%|3 | 176/4556 [00:05<02:05, 34.96it/s]
4%|3 | 180/4556 [00:05<02:06, 34.58it/s]
4%|4 | 184/4556 [00:05<02:06, 34.63it/s]
4%|4 | 188/4556 [00:05<02:05, 34.76it/s]
4%|4 | 192/4556 [00:05<02:08, 33.88it/s]
4%|4 | 196/4556 [00:05<02:08, 34.00it/s]
4%|4 | 200/4556 [00:06<02:06, 34.34it/s]
4%|4 | 204/4556 [00:06<02:05, 34.56it/s]
5%|4 | 208/4556 [00:06<02:05, 34.69it/s]
5%|4 | 212/4556 [00:06<02:04, 34.82it/s]
5%|4 | 216/4556 [00:06<02:05, 34.45it/s]
5%|4 | 220/4556 [00:06<02:04, 34.72it/s]
5%|4 | 224/4556 [00:06<02:04, 34.82it/s]
5%|5 | 228/4556 [00:06<02:03, 34.91it/s]
5%|5 | 232/4556 [00:06<02:03, 35.02it/s]
5%|5 | 236/4556 [00:07<02:03, 35.00it/s]
5%|5 | 240/4556 [00:07<02:03, 35.05it/s]
5%|5 | 244/4556 [00:07<02:02, 35.10it/s]
5%|5 | 248/4556 [00:07<02:02, 35.06it/s]
6%|5 | 252/4556 [00:07<02:04, 34.60it/s]
6%|5 | 256/4556 [00:07<02:03, 34.73it/s]
6%|5 | 260/4556 [00:07<02:03, 34.79it/s]
6%|5 | 264/4556 [00:07<02:03, 34.87it/s]
6%|5 | 268/4556 [00:08<02:02, 34.92it/s]
6%|5 | 272/4556 [00:08<02:02, 35.02it/s]
6%|6 | 276/4556 [00:08<02:01, 35.12it/s]
6%|6 | 280/4556 [00:08<02:01, 35.09it/s]
6%|6 | 284/4556 [00:08<02:01, 35.12it/s]
6%|6 | 288/4556 [00:08<02:02, 34.90it/s]
6%|6 | 292/4556 [00:08<02:03, 34.57it/s]
6%|6 | 296/4556 [00:08<02:02, 34.66it/s]
7%|6 | 300/4556 [00:08<02:02, 34.84it/s]
7%|6 | 304/4556 [00:09<02:02, 34.77it/s]
7%|6 | 308/4556 [00:09<02:01, 34.88it/s]
7%|6 | 312/4556 [00:09<02:01, 34.93it/s]
7%|6 | 316/4556 [00:09<02:01, 34.98it/s]
7%|7 | 320/4556 [00:09<02:01, 34.97it/s]
7%|7 | 324/4556 [00:09<02:01, 34.89it/s]
7%|7 | 328/4556 [00:09<02:02, 34.43it/s]
7%|7 | 332/4556 [00:09<02:02, 34.58it/s]
7%|7 | 336/4556 [00:09<02:01, 34.64it/s]
7%|7 | 340/4556 [00:10<02:01, 34.71it/s]
8%|7 | 344/4556 [00:10<02:01, 34.68it/s]
8%|7 | 348/4556 [00:10<02:29, 28.18it/s]
8%|7 | 352/4556 [00:10<02:20, 29.90it/s]
8%|7 | 356/4556 [00:10<02:14, 31.25it/s]
8%|7 | 360/4556 [00:10<02:12, 31.78it/s]
8%|7 | 364/4556 [00:10<02:08, 32.63it/s]
8%|8 | 368/4556 [00:10<02:05, 33.25it/s]
8%|8 | 372/4556 [00:11<02:04, 33.69it/s]
8%|8 | 376/4556 [00:11<02:02, 34.05it/s]
8%|8 | 380/4556 [00:11<02:01, 34.27it/s]
8%|8 | 384/4556 [00:11<02:01, 34.45it/s]
9%|8 | 388/4556 [00:11<02:00, 34.59it/s]
9%|8 | 392/4556 [00:11<02:04, 33.52it/s]
9%|8 | 396/4556 [00:11<02:04, 33.37it/s]
9%|8 | 400/4556 [00:11<02:03, 33.78it/s]
9%|8 | 404/4556 [00:12<02:01, 34.17it/s]
9%|8 | 408/4556 [00:12<02:00, 34.35it/s]
9%|9 | 412/4556 [00:12<02:00, 34.49it/s]
9%|9 | 416/4556 [00:12<01:59, 34.60it/s]
9%|9 | 420/4556 [00:12<01:59, 34.65it/s]
9%|9 | 424/4556 [00:12<01:59, 34.68it/s]
9%|9 | 428/4556 [00:12<01:59, 34.65it/s]
9%|9 | 432/4556 [00:12<02:00, 34.15it/s]
10%|9 | 436/4556 [00:12<02:00, 34.33it/s]
10%|9 | 440/4556 [00:13<01:59, 34.37it/s]
10%|9 | 444/4556 [00:13<02:08, 31.99it/s]
10%|9 | 448/4556 [00:13<02:05, 32.75it/s]
10%|9 | 452/4556 [00:13<02:03, 33.29it/s]
10%|# | 456/4556 [00:13<02:01, 33.70it/s]
10%|# | 460/4556 [00:13<02:00, 33.93it/s]
10%|# | 464/4556 [00:13<02:01, 33.71it/s]
10%|# | 468/4556 [00:13<02:00, 34.06it/s]
10%|# | 472/4556 [00:14<01:59, 34.28it/s]
10%|# | 476/4556 [00:14<01:58, 34.42it/s]
11%|# | 480/4556 [00:14<01:57, 34.56it/s]
11%|# | 484/4556 [00:14<01:57, 34.59it/s]
11%|# | 488/4556 [00:14<01:57, 34.59it/s]
11%|# | 492/4556 [00:14<01:57, 34.65it/s]
11%|# | 496/4556 [00:14<01:57, 34.68it/s]
11%|# | 500/4556 [00:14<01:58, 34.24it/s]
11%|#1 | 504/4556 [00:14<01:57, 34.39it/s]
11%|#1 | 508/4556 [00:15<01:57, 34.47it/s]
11%|#1 | 512/4556 [00:15<01:56, 34.59it/s]
11%|#1 | 516/4556 [00:15<01:56, 34.68it/s]
11%|#1 | 520/4556 [00:15<01:56, 34.75it/s]
12%|#1 | 524/4556 [00:15<01:55, 34.80it/s]
12%|#1 | 528/4556 [00:15<01:55, 34.77it/s]
12%|#1 | 532/4556 [00:15<01:55, 34.74it/s]
12%|#1 | 536/4556 [00:15<01:57, 34.33it/s]
12%|#1 | 540/4556 [00:15<01:56, 34.45it/s]
12%|#1 | 544/4556 [00:16<01:56, 34.56it/s]
12%|#2 | 548/4556 [00:16<01:55, 34.65it/s]
12%|#2 | 552/4556 [00:16<01:55, 34.74it/s]
12%|#2 | 556/4556 [00:16<01:54, 34.81it/s]
12%|#2 | 560/4556 [00:16<01:54, 34.92it/s]
12%|#2 | 564/4556 [00:16<01:54, 34.90it/s]
12%|#2 | 568/4556 [00:16<01:54, 34.89it/s]
13%|#2 | 572/4556 [00:16<01:56, 34.32it/s]
13%|#2 | 576/4556 [00:17<01:55, 34.58it/s]
13%|#2 | 580/4556 [00:17<01:54, 34.66it/s]
13%|#2 | 584/4556 [00:17<01:54, 34.79it/s]
13%|#2 | 588/4556 [00:17<01:53, 34.84it/s]
13%|#2 | 592/4556 [00:17<01:53, 34.86it/s]
13%|#3 | 596/4556 [00:17<01:53, 34.86it/s]
13%|#3 | 600/4556 [00:17<01:53, 34.85it/s]
13%|#3 | 604/4556 [00:17<01:53, 34.80it/s]
13%|#3 | 608/4556 [00:17<01:55, 34.33it/s]
13%|#3 | 612/4556 [00:18<01:54, 34.52it/s]
14%|#3 | 616/4556 [00:18<01:53, 34.61it/s]
14%|#3 | 620/4556 [00:18<01:53, 34.73it/s]
14%|#3 | 624/4556 [00:18<01:53, 34.73it/s]
14%|#3 | 628/4556 [00:18<01:53, 34.75it/s]
14%|#3 | 632/4556 [00:18<01:52, 34.76it/s]
14%|#3 | 636/4556 [00:18<01:52, 34.81it/s]
14%|#4 | 640/4556 [00:18<01:52, 34.83it/s]
14%|#4 | 644/4556 [00:18<01:56, 33.69it/s]
14%|#4 | 648/4556 [00:19<01:54, 34.11it/s]
14%|#4 | 652/4556 [00:19<01:53, 34.30it/s]
14%|#4 | 656/4556 [00:19<01:53, 34.49it/s]
14%|#4 | 660/4556 [00:19<01:52, 34.49it/s]
15%|#4 | 664/4556 [00:19<01:52, 34.58it/s]
15%|#4 | 668/4556 [00:19<01:52, 34.63it/s]
15%|#4 | 672/4556 [00:19<01:52, 34.55it/s]
15%|#4 | 676/4556 [00:19<01:51, 34.65it/s]
15%|#4 | 680/4556 [00:20<01:55, 33.62it/s]
15%|#5 | 684/4556 [00:20<01:54, 33.93it/s]
15%|#5 | 688/4556 [00:20<01:53, 34.13it/s]
15%|#5 | 692/4556 [00:20<01:52, 34.32it/s]
15%|#5 | 696/4556 [00:20<01:52, 34.46it/s]
15%|#5 | 700/4556 [00:20<01:51, 34.58it/s]
15%|#5 | 704/4556 [00:20<01:51, 34.66it/s]
16%|#5 | 708/4556 [00:20<01:51, 34.65it/s]
16%|#5 | 712/4556 [00:20<01:50, 34.73it/s]
16%|#5 | 716/4556 [00:21<01:53, 33.72it/s]
16%|#5 | 720/4556 [00:21<01:52, 34.01it/s]
16%|#5 | 724/4556 [00:21<01:51, 34.31it/s]
16%|#5 | 728/4556 [00:21<01:51, 34.48it/s]
16%|#6 | 732/4556 [00:21<01:50, 34.56it/s]
16%|#6 | 736/4556 [00:21<01:50, 34.58it/s]
16%|#6 | 740/4556 [00:21<01:50, 34.60it/s]
16%|#6 | 744/4556 [00:21<01:50, 34.58it/s]
16%|#6 | 748/4556 [00:22<01:50, 34.54it/s]
17%|#6 | 752/4556 [00:22<01:53, 33.54it/s]
17%|#6 | 756/4556 [00:22<01:51, 33.94it/s]
17%|#6 | 760/4556 [00:22<01:51, 34.14it/s]
17%|#6 | 764/4556 [00:22<01:53, 33.37it/s]
17%|#6 | 768/4556 [00:22<01:53, 33.26it/s]
17%|#6 | 772/4556 [00:22<01:52, 33.76it/s]
17%|#7 | 776/4556 [00:22<01:51, 34.04it/s]
17%|#7 | 780/4556 [00:22<01:50, 34.26it/s]
17%|#7 | 784/4556 [00:23<01:49, 34.41it/s]
17%|#7 | 788/4556 [00:23<01:52, 33.50it/s]
17%|#7 | 792/4556 [00:23<01:51, 33.85it/s]
17%|#7 | 796/4556 [00:23<01:50, 34.09it/s]
18%|#7 | 800/4556 [00:23<01:49, 34.29it/s]
18%|#7 | 804/4556 [00:23<01:49, 34.40it/s]
18%|#7 | 808/4556 [00:23<01:48, 34.45it/s]
18%|#7 | 812/4556 [00:23<01:48, 34.57it/s]
18%|#7 | 816/4556 [00:24<01:47, 34.63it/s]
18%|#7 | 820/4556 [00:24<01:47, 34.63it/s]
18%|#8 | 824/4556 [00:24<01:50, 33.64it/s]
18%|#8 | 828/4556 [00:24<01:49, 33.97it/s]
18%|#8 | 832/4556 [00:24<01:49, 34.16it/s]
18%|#8 | 836/4556 [00:24<01:48, 34.37it/s]
18%|#8 | 840/4556 [00:24<01:47, 34.48it/s]
19%|#8 | 844/4556 [00:24<01:47, 34.55it/s]
19%|#8 | 848/4556 [00:24<01:48, 34.14it/s]
19%|#8 | 852/4556 [00:25<01:51, 33.30it/s]
19%|#8 | 856/4556 [00:25<01:49, 33.74it/s]
19%|#8 | 860/4556 [00:25<01:51, 33.06it/s]
19%|#8 | 864/4556 [00:25<01:50, 33.51it/s]
19%|#9 | 868/4556 [00:25<01:48, 33.87it/s]
19%|#9 | 872/4556 [00:25<01:48, 34.07it/s]
19%|#9 | 876/4556 [00:25<01:47, 34.22it/s]
19%|#9 | 880/4556 [00:25<01:46, 34.38it/s]
19%|#9 | 884/4556 [00:26<01:46, 34.55it/s]
19%|#9 | 888/4556 [00:26<01:45, 34.66it/s]
20%|#9 | 892/4556 [00:26<01:45, 34.65it/s]
20%|#9 | 896/4556 [00:26<01:48, 33.64it/s]
20%|#9 | 900/4556 [00:26<01:47, 34.01it/s]
20%|#9 | 904/4556 [00:26<01:46, 34.29it/s]
20%|#9 | 908/4556 [00:26<01:45, 34.47it/s]
20%|## | 912/4556 [00:26<01:45, 34.47it/s]
20%|## | 916/4556 [00:26<01:45, 34.55it/s]
20%|## | 920/4556 [00:27<01:45, 34.57it/s]
20%|## | 924/4556 [00:27<01:44, 34.66it/s]
20%|## | 928/4556 [00:27<01:44, 34.73it/s]
20%|## | 932/4556 [00:27<01:47, 33.67it/s]
21%|## | 936/4556 [00:27<01:46, 33.94it/s]
21%|## | 940/4556 [00:27<01:45, 34.15it/s]
21%|## | 944/4556 [00:27<01:45, 34.34it/s]
21%|## | 948/4556 [00:27<01:44, 34.44it/s]
21%|## | 952/4556 [00:27<01:44, 34.51it/s]
21%|## | 956/4556 [00:28<01:44, 34.51it/s]
21%|##1 | 960/4556 [00:28<01:43, 34.58it/s]
21%|##1 | 964/4556 [00:28<01:44, 34.40it/s]
21%|##1 | 968/4556 [00:28<01:46, 33.61it/s]
21%|##1 | 972/4556 [00:28<01:45, 33.90it/s]
21%|##1 | 976/4556 [00:28<01:45, 34.06it/s]
22%|##1 | 980/4556 [00:28<01:44, 34.27it/s]
22%|##1 | 984/4556 [00:28<01:43, 34.40it/s]
22%|##1 | 988/4556 [00:29<01:43, 34.54it/s]
22%|##1 | 992/4556 [00:29<01:44, 34.13it/s]
22%|##1 | 996/4556 [00:29<01:46, 33.42it/s]
22%|##1 | 1000/4556 [00:29<01:48, 32.88it/s]
22%|##2 | 1004/4556 [00:29<01:46, 33.30it/s]
22%|##2 | 1008/4556 [00:29<01:45, 33.70it/s]
22%|##2 | 1012/4556 [00:29<01:44, 33.92it/s]
22%|##2 | 1016/4556 [00:29<01:43, 34.21it/s]
22%|##2 | 1020/4556 [00:29<01:42, 34.34it/s]
22%|##2 | 1024/4556 [00:30<01:42, 34.41it/s]
23%|##2 | 1028/4556 [00:30<01:42, 34.54it/s]
23%|##2 | 1032/4556 [00:30<01:42, 34.52it/s]
23%|##2 | 1036/4556 [00:30<01:44, 33.55it/s]
23%|##2 | 1040/4556 [00:30<01:43, 33.87it/s]
23%|##2 | 1044/4556 [00:30<01:43, 34.07it/s]
23%|##3 | 1048/4556 [00:30<01:42, 34.18it/s]
23%|##3 | 1052/4556 [00:30<01:42, 34.35it/s]
23%|##3 | 1056/4556 [00:31<01:41, 34.39it/s]
23%|##3 | 1060/4556 [00:31<01:41, 34.57it/s]
23%|##3 | 1064/4556 [00:31<01:48, 32.09it/s]
23%|##3 | 1068/4556 [00:31<01:46, 32.87it/s]
24%|##3 | 1072/4556 [00:31<01:47, 32.41it/s]
24%|##3 | 1076/4556 [00:31<01:45, 33.08it/s]
24%|##3 | 1080/4556 [00:31<02:06, 27.58it/s]
24%|##3 | 1084/4556 [00:31<01:58, 29.23it/s]
24%|##3 | 1088/4556 [00:32<01:53, 30.55it/s]
24%|##3 | 1092/4556 [00:32<01:49, 31.66it/s]
24%|##4 | 1096/4556 [00:32<01:46, 32.55it/s]
24%|##4 | 1100/4556 [00:32<01:44, 33.16it/s]
24%|##4 | 1104/4556 [00:32<01:46, 32.52it/s]
24%|##4 | 1108/4556 [00:32<01:44, 33.12it/s]
24%|##4 | 1112/4556 [00:32<01:42, 33.57it/s]
24%|##4 | 1116/4556 [00:32<01:41, 33.83it/s]
25%|##4 | 1120/4556 [00:33<01:41, 33.96it/s]
25%|##4 | 1124/4556 [00:33<01:40, 34.18it/s]
25%|##4 | 1128/4556 [00:33<01:40, 34.28it/s]
25%|##4 | 1132/4556 [00:33<01:39, 34.31it/s]
25%|##4 | 1136/4556 [00:33<01:39, 34.37it/s]
25%|##5 | 1140/4556 [00:33<01:42, 33.36it/s]
25%|##5 | 1144/4556 [00:33<01:41, 33.70it/s]
25%|##5 | 1148/4556 [00:33<01:40, 33.90it/s]
25%|##5 | 1152/4556 [00:33<01:39, 34.09it/s]
25%|##5 | 1156/4556 [00:34<01:39, 34.23it/s]
25%|##5 | 1160/4556 [00:34<01:38, 34.32it/s]
26%|##5 | 1164/4556 [00:34<01:38, 34.38it/s]
26%|##5 | 1168/4556 [00:34<01:38, 34.44it/s]
26%|##5 | 1172/4556 [00:34<01:38, 34.46it/s]
26%|##5 | 1176/4556 [00:34<01:41, 33.41it/s]
26%|##5 | 1180/4556 [00:34<01:40, 33.68it/s]
26%|##5 | 1184/4556 [00:34<01:39, 33.98it/s]
26%|##6 | 1188/4556 [00:35<01:38, 34.16it/s]
26%|##6 | 1192/4556 [00:35<01:37, 34.37it/s]
26%|##6 | 1196/4556 [00:35<01:37, 34.38it/s]
26%|##6 | 1200/4556 [00:35<01:37, 34.48it/s]
26%|##6 | 1204/4556 [00:35<01:37, 34.54it/s]
27%|##6 | 1208/4556 [00:35<01:37, 34.51it/s]
27%|##6 | 1212/4556 [00:35<01:39, 33.54it/s]
27%|##6 | 1216/4556 [00:35<01:38, 33.87it/s]
27%|##6 | 1220/4556 [00:35<01:37, 34.09it/s]
27%|##6 | 1224/4556 [00:36<01:37, 34.29it/s]
27%|##6 | 1228/4556 [00:36<01:36, 34.35it/s]
27%|##7 | 1232/4556 [00:36<01:36, 34.39it/s]
27%|##7 | 1236/4556 [00:36<01:36, 34.40it/s]
27%|##7 | 1240/4556 [00:36<01:36, 34.44it/s]
27%|##7 | 1244/4556 [00:36<01:36, 34.47it/s]
27%|##7 | 1248/4556 [00:36<01:38, 33.53it/s]
27%|##7 | 1252/4556 [00:36<01:37, 33.83it/s]
28%|##7 | 1256/4556 [00:37<01:36, 34.09it/s]
28%|##7 | 1260/4556 [00:37<01:36, 34.20it/s]
28%|##7 | 1264/4556 [00:37<01:35, 34.32it/s]
28%|##7 | 1268/4556 [00:37<01:35, 34.39it/s]
28%|##7 | 1272/4556 [00:37<01:35, 34.46it/s]
28%|##8 | 1276/4556 [00:37<01:35, 34.51it/s]
28%|##8 | 1280/4556 [00:37<01:35, 34.44it/s]
28%|##8 | 1284/4556 [00:37<01:37, 33.46it/s]
28%|##8 | 1288/4556 [00:37<01:36, 33.73it/s]
28%|##8 | 1292/4556 [00:38<01:36, 33.99it/s]
28%|##8 | 1296/4556 [00:38<01:35, 34.16it/s]
29%|##8 | 1300/4556 [00:38<01:35, 34.25it/s]
29%|##8 | 1304/4556 [00:38<01:34, 34.43it/s]
29%|##8 | 1308/4556 [00:38<01:34, 34.47it/s]
29%|##8 | 1312/4556 [00:38<01:34, 34.45it/s]
29%|##8 | 1316/4556 [00:38<01:35, 33.92it/s]
29%|##8 | 1320/4556 [00:38<01:36, 33.48it/s]
29%|##9 | 1324/4556 [00:39<01:35, 33.78it/s]
29%|##9 | 1328/4556 [00:39<01:35, 33.95it/s]
29%|##9 | 1332/4556 [00:39<01:34, 34.18it/s]
29%|##9 | 1336/4556 [00:39<01:33, 34.34it/s]
29%|##9 | 1340/4556 [00:39<01:33, 34.34it/s]
29%|##9 | 1344/4556 [00:39<01:33, 34.36it/s]
30%|##9 | 1348/4556 [00:39<01:33, 34.33it/s]
30%|##9 | 1352/4556 [00:39<01:36, 33.24it/s]
30%|##9 | 1356/4556 [00:39<01:35, 33.65it/s]
30%|##9 | 1360/4556 [00:40<01:34, 33.91it/s]
30%|##9 | 1364/4556 [00:40<01:33, 34.11it/s]
30%|### | 1368/4556 [00:40<01:36, 33.04it/s]
30%|### | 1372/4556 [00:40<01:35, 33.49it/s]
30%|### | 1376/4556 [00:40<01:34, 33.73it/s]
30%|### | 1380/4556 [00:40<01:33, 34.01it/s]
30%|### | 1384/4556 [00:40<01:32, 34.15it/s]
30%|### | 1388/4556 [00:40<01:35, 33.29it/s]
31%|### | 1392/4556 [00:41<01:34, 33.60it/s]
31%|### | 1396/4556 [00:41<01:33, 33.86it/s]
31%|### | 1400/4556 [00:41<01:32, 34.00it/s]
31%|### | 1404/4556 [00:41<01:32, 34.19it/s]
31%|### | 1408/4556 [00:41<01:31, 34.23it/s]
31%|### | 1412/4556 [00:41<01:31, 34.31it/s]
31%|###1 | 1416/4556 [00:41<01:31, 34.35it/s]
31%|###1 | 1420/4556 [00:41<01:31, 34.31it/s]
31%|###1 | 1424/4556 [00:41<01:33, 33.40it/s]
31%|###1 | 1428/4556 [00:42<01:32, 33.69it/s]
31%|###1 | 1432/4556 [00:42<01:32, 33.93it/s]
32%|###1 | 1436/4556 [00:42<01:31, 34.02it/s]
32%|###1 | 1440/4556 [00:42<01:31, 34.14it/s]
32%|###1 | 1444/4556 [00:42<01:31, 34.13it/s]
32%|###1 | 1448/4556 [00:42<01:30, 34.17it/s]
32%|###1 | 1452/4556 [00:42<01:30, 34.19it/s]
32%|###1 | 1456/4556 [00:42<01:30, 34.31it/s]
32%|###2 | 1460/4556 [00:43<01:32, 33.32it/s]
32%|###2 | 1464/4556 [00:43<01:31, 33.71it/s]
32%|###2 | 1468/4556 [00:43<01:31, 33.93it/s]
32%|###2 | 1472/4556 [00:43<01:30, 33.98it/s]
32%|###2 | 1476/4556 [00:43<01:31, 33.49it/s]
32%|###2 | 1480/4556 [00:43<01:32, 33.34it/s]
33%|###2 | 1484/4556 [00:43<01:31, 33.61it/s]
33%|###2 | 1488/4556 [00:43<01:30, 33.81it/s]
33%|###2 | 1492/4556 [00:43<01:30, 33.67it/s]
33%|###2 | 1496/4556 [00:44<01:32, 33.01it/s]
33%|###2 | 1500/4556 [00:44<01:31, 33.52it/s]
33%|###3 | 1504/4556 [00:44<01:30, 33.82it/s]
33%|###3 | 1508/4556 [00:44<01:29, 34.05it/s]
33%|###3 | 1512/4556 [00:44<01:29, 34.16it/s]
33%|###3 | 1516/4556 [00:44<01:28, 34.17it/s]
33%|###3 | 1520/4556 [00:44<01:28, 34.31it/s]
33%|###3 | 1524/4556 [00:44<01:28, 34.30it/s]
34%|###3 | 1528/4556 [00:45<01:30, 33.62it/s]
34%|###3 | 1532/4556 [00:45<01:30, 33.50it/s]
34%|###3 | 1536/4556 [00:45<01:29, 33.80it/s]
34%|###3 | 1540/4556 [00:45<01:28, 33.98it/s]
34%|###3 | 1544/4556 [00:45<01:28, 34.12it/s]
34%|###3 | 1548/4556 [00:45<01:27, 34.19it/s]
34%|###4 | 1552/4556 [00:45<01:27, 34.30it/s]
34%|###4 | 1556/4556 [00:45<01:27, 34.29it/s]
34%|###4 | 1560/4556 [00:45<01:27, 34.30it/s]
34%|###4 | 1564/4556 [00:46<01:29, 33.26it/s]
34%|###4 | 1568/4556 [00:46<01:31, 32.72it/s]
35%|###4 | 1572/4556 [00:46<01:34, 31.66it/s]
35%|###4 | 1576/4556 [00:46<01:31, 32.46it/s]
35%|###4 | 1580/4556 [00:46<01:30, 33.03it/s]
35%|###4 | 1584/4556 [00:46<01:28, 33.46it/s]
35%|###4 | 1588/4556 [00:46<01:27, 33.79it/s]
35%|###4 | 1592/4556 [00:46<01:27, 33.98it/s]
35%|###5 | 1596/4556 [00:47<01:26, 34.12it/s]
35%|###5 | 1600/4556 [00:47<01:29, 33.14it/s]
35%|###5 | 1604/4556 [00:47<01:28, 33.49it/s]
35%|###5 | 1608/4556 [00:47<01:27, 33.78it/s]
35%|###5 | 1612/4556 [00:47<01:26, 33.93it/s]
35%|###5 | 1616/4556 [00:47<01:26, 34.07it/s]
36%|###5 | 1620/4556 [00:47<01:25, 34.21it/s]
36%|###5 | 1624/4556 [00:47<01:25, 34.24it/s]
36%|###5 | 1628/4556 [00:47<01:25, 34.29it/s]
36%|###5 | 1632/4556 [00:48<01:25, 34.32it/s]
36%|###5 | 1636/4556 [00:48<01:27, 33.37it/s]
36%|###5 | 1640/4556 [00:48<01:26, 33.67it/s]
36%|###6 | 1644/4556 [00:48<01:25, 33.89it/s]
36%|###6 | 1648/4556 [00:48<01:25, 33.93it/s]
36%|###6 | 1652/4556 [00:48<01:25, 34.07it/s]
36%|###6 | 1656/4556 [00:48<01:24, 34.15it/s]
36%|###6 | 1660/4556 [00:48<01:24, 34.24it/s]
37%|###6 | 1664/4556 [00:49<01:24, 34.33it/s]
37%|###6 | 1668/4556 [00:49<01:23, 34.44it/s]
37%|###6 | 1672/4556 [00:49<01:26, 33.42it/s]
37%|###6 | 1676/4556 [00:49<01:25, 33.63it/s]
37%|###6 | 1680/4556 [00:49<01:24, 33.93it/s]
37%|###6 | 1684/4556 [00:49<01:24, 34.12it/s]
37%|###7 | 1688/4556 [00:49<01:23, 34.24it/s]
37%|###7 | 1692/4556 [00:49<01:23, 34.35it/s]
37%|###7 | 1696/4556 [00:49<01:23, 34.35it/s]
37%|###7 | 1700/4556 [00:50<01:22, 34.46it/s]
37%|###7 | 1704/4556 [00:50<01:23, 34.31it/s]
37%|###7 | 1708/4556 [00:50<01:24, 33.56it/s]
38%|###7 | 1712/4556 [00:50<01:24, 33.80it/s]
38%|###7 | 1716/4556 [00:50<01:23, 33.94it/s]
38%|###7 | 1720/4556 [00:50<01:23, 34.08it/s]
38%|###7 | 1724/4556 [00:50<01:22, 34.21it/s]
38%|###7 | 1728/4556 [00:50<01:22, 34.30it/s]
38%|###8 | 1732/4556 [00:51<01:22, 34.33it/s]
38%|###8 | 1736/4556 [00:51<01:21, 34.42it/s]
38%|###8 | 1740/4556 [00:51<01:23, 33.74it/s]
38%|###8 | 1744/4556 [00:51<01:23, 33.61it/s]
38%|###8 | 1748/4556 [00:51<01:22, 33.85it/s]
38%|###8 | 1752/4556 [00:51<01:22, 34.05it/s]
39%|###8 | 1756/4556 [00:51<01:21, 34.22it/s]
39%|###8 | 1760/4556 [00:51<01:21, 34.24it/s]
39%|###8 | 1764/4556 [00:51<01:21, 34.35it/s]
39%|###8 | 1768/4556 [00:52<01:21, 34.25it/s]
39%|###8 | 1772/4556 [00:52<01:21, 34.34it/s]
39%|###8 | 1776/4556 [00:52<01:23, 33.34it/s]
39%|###9 | 1780/4556 [00:52<01:22, 33.68it/s]
39%|###9 | 1784/4556 [00:52<01:21, 33.86it/s]
39%|###9 | 1788/4556 [00:52<01:21, 34.05it/s]
39%|###9 | 1792/4556 [00:52<01:20, 34.19it/s]
39%|###9 | 1796/4556 [00:52<01:20, 34.25it/s]
40%|###9 | 1800/4556 [00:53<01:20, 34.16it/s]
40%|###9 | 1804/4556 [00:53<01:20, 34.12it/s]
40%|###9 | 1808/4556 [00:53<01:20, 34.25it/s]
40%|###9 | 1812/4556 [00:53<01:22, 33.26it/s]
40%|###9 | 1816/4556 [00:53<01:21, 33.64it/s]
40%|###9 | 1820/4556 [00:53<01:20, 33.84it/s]
40%|#### | 1824/4556 [00:53<01:37, 27.92it/s]
40%|#### | 1828/4556 [00:53<01:32, 29.60it/s]
40%|#### | 1832/4556 [00:54<01:28, 30.89it/s]
40%|#### | 1836/4556 [00:54<01:25, 31.84it/s]
40%|#### | 1840/4556 [00:54<01:23, 32.58it/s]
40%|#### | 1844/4556 [00:54<01:24, 32.17it/s]
41%|#### | 1848/4556 [00:54<01:22, 32.81it/s]
41%|#### | 1852/4556 [00:54<01:21, 33.25it/s]
41%|#### | 1856/4556 [00:54<01:20, 33.63it/s]
41%|#### | 1860/4556 [00:54<01:19, 33.84it/s]
41%|#### | 1864/4556 [00:55<01:19, 33.97it/s]
41%|####1 | 1868/4556 [00:55<01:18, 34.08it/s]
41%|####1 | 1872/4556 [00:55<01:18, 34.17it/s]
41%|####1 | 1876/4556 [00:55<01:18, 34.27it/s]
41%|####1 | 1880/4556 [00:55<01:20, 33.19it/s]
41%|####1 | 1884/4556 [00:55<01:19, 33.51it/s]
41%|####1 | 1888/4556 [00:55<01:19, 33.76it/s]
42%|####1 | 1892/4556 [00:55<01:18, 33.93it/s]
42%|####1 | 1896/4556 [00:55<01:18, 34.08it/s]
42%|####1 | 1900/4556 [00:56<01:17, 34.17it/s]
42%|####1 | 1904/4556 [00:56<01:17, 34.27it/s]
42%|####1 | 1908/4556 [00:56<01:17, 34.29it/s]
42%|####1 | 1912/4556 [00:56<01:17, 34.32it/s]
42%|####2 | 1916/4556 [00:56<01:19, 33.32it/s]
42%|####2 | 1920/4556 [00:56<01:18, 33.57it/s]
42%|####2 | 1924/4556 [00:56<01:17, 33.75it/s]
42%|####2 | 1928/4556 [00:56<01:17, 34.00it/s]
42%|####2 | 1932/4556 [00:57<01:17, 34.05it/s]
42%|####2 | 1936/4556 [00:57<01:16, 34.15it/s]
43%|####2 | 1940/4556 [00:57<01:16, 34.25it/s]
43%|####2 | 1944/4556 [00:57<01:16, 34.31it/s]
43%|####2 | 1948/4556 [00:57<01:16, 34.26it/s]
43%|####2 | 1952/4556 [00:57<01:18, 33.25it/s]
43%|####2 | 1956/4556 [00:57<01:17, 33.56it/s]
43%|####3 | 1960/4556 [00:57<01:16, 33.80it/s]
43%|####3 | 1964/4556 [00:57<01:16, 33.97it/s]
43%|####3 | 1968/4556 [00:58<01:16, 34.03it/s]
43%|####3 | 1972/4556 [00:58<01:15, 34.16it/s]
43%|####3 | 1976/4556 [00:58<01:15, 34.25it/s]
43%|####3 | 1980/4556 [00:58<01:15, 34.28it/s]
44%|####3 | 1984/4556 [00:58<01:16, 33.42it/s]
44%|####3 | 1988/4556 [00:58<01:18, 32.59it/s]
44%|####3 | 1992/4556 [00:58<01:17, 33.06it/s]
44%|####3 | 1996/4556 [00:58<01:16, 33.46it/s]
44%|####3 | 2000/4556 [00:59<01:15, 33.74it/s]
44%|####3 | 2004/4556 [00:59<01:15, 33.84it/s]
44%|####4 | 2008/4556 [00:59<01:14, 34.01it/s]
44%|####4 | 2012/4556 [00:59<01:14, 34.07it/s]
44%|####4 | 2016/4556 [00:59<01:14, 34.19it/s]
44%|####4 | 2020/4556 [00:59<01:14, 34.25it/s]
44%|####4 | 2024/4556 [00:59<01:14, 33.80it/s]
45%|####4 | 2028/4556 [00:59<01:14, 33.94it/s]
45%|####4 | 2032/4556 [00:59<01:14, 34.02it/s]
45%|####4 | 2036/4556 [01:00<01:13, 34.11it/s]
45%|####4 | 2040/4556 [01:00<01:13, 34.20it/s]
45%|####4 | 2044/4556 [01:00<01:13, 34.29it/s]
45%|####4 | 2048/4556 [01:00<01:13, 34.32it/s]
45%|####5 | 2052/4556 [01:00<01:13, 34.27it/s]
45%|####5 | 2056/4556 [01:00<01:13, 34.13it/s]
45%|####5 | 2060/4556 [01:00<01:13, 33.76it/s]
45%|####5 | 2064/4556 [01:00<01:13, 33.99it/s]
45%|####5 | 2068/4556 [01:01<01:12, 34.09it/s]
45%|####5 | 2072/4556 [01:01<01:12, 34.21it/s]
46%|####5 | 2076/4556 [01:01<01:12, 34.30it/s]
46%|####5 | 2080/4556 [01:01<01:12, 34.32it/s]
46%|####5 | 2084/4556 [01:01<01:11, 34.34it/s]
46%|####5 | 2088/4556 [01:01<01:11, 34.35it/s]
46%|####5 | 2092/4556 [01:01<01:12, 33.93it/s]
46%|####6 | 2096/4556 [01:01<01:12, 34.03it/s]
46%|####6 | 2100/4556 [01:01<01:11, 34.15it/s]
46%|####6 | 2104/4556 [01:02<01:11, 34.21it/s]
46%|####6 | 2108/4556 [01:02<01:11, 34.28it/s]
46%|####6 | 2112/4556 [01:02<01:11, 34.33it/s]
46%|####6 | 2116/4556 [01:02<01:10, 34.40it/s]
47%|####6 | 2120/4556 [01:02<01:10, 34.42it/s]
47%|####6 | 2124/4556 [01:02<01:10, 34.44it/s]
47%|####6 | 2128/4556 [01:02<01:11, 33.94it/s]
47%|####6 | 2132/4556 [01:02<01:11, 34.08it/s]
47%|####6 | 2136/4556 [01:03<01:10, 34.20it/s]
47%|####6 | 2140/4556 [01:03<01:10, 34.19it/s]
47%|####7 | 2144/4556 [01:03<01:10, 34.25it/s]
47%|####7 | 2148/4556 [01:03<01:10, 34.25it/s]
47%|####7 | 2152/4556 [01:03<01:10, 34.25it/s]
47%|####7 | 2156/4556 [01:03<01:10, 34.26it/s]
47%|####7 | 2160/4556 [01:03<01:09, 34.26it/s]
47%|####7 | 2164/4556 [01:03<01:10, 33.82it/s]
48%|####7 | 2168/4556 [01:03<01:10, 33.92it/s]
48%|####7 | 2172/4556 [01:04<01:10, 34.05it/s]
48%|####7 | 2176/4556 [01:04<01:09, 34.16it/s]
48%|####7 | 2180/4556 [01:04<01:09, 34.23it/s]
48%|####7 | 2184/4556 [01:04<01:09, 34.23it/s]
48%|####8 | 2188/4556 [01:04<01:08, 34.33it/s]
48%|####8 | 2192/4556 [01:04<01:08, 34.26it/s]
48%|####8 | 2196/4556 [01:04<01:08, 34.28it/s]
48%|####8 | 2200/4556 [01:04<01:09, 33.75it/s]
48%|####8 | 2204/4556 [01:05<01:09, 33.94it/s]
48%|####8 | 2208/4556 [01:05<01:08, 34.09it/s]
49%|####8 | 2212/4556 [01:05<01:08, 34.22it/s]
49%|####8 | 2216/4556 [01:05<01:08, 34.25it/s]
49%|####8 | 2220/4556 [01:05<01:08, 34.35it/s]
49%|####8 | 2224/4556 [01:05<01:07, 34.39it/s]
49%|####8 | 2228/4556 [01:05<01:07, 34.37it/s]
49%|####8 | 2232/4556 [01:05<01:08, 34.13it/s]
49%|####9 | 2236/4556 [01:05<01:08, 33.75it/s]
49%|####9 | 2240/4556 [01:06<01:08, 34.01it/s]
49%|####9 | 2244/4556 [01:06<01:07, 34.16it/s]
49%|####9 | 2248/4556 [01:06<01:07, 34.16it/s]
49%|####9 | 2252/4556 [01:06<01:07, 34.16it/s]
50%|####9 | 2256/4556 [01:06<01:07, 34.17it/s]
50%|####9 | 2260/4556 [01:06<01:07, 34.23it/s]
50%|####9 | 2264/4556 [01:06<01:06, 34.27it/s]
50%|####9 | 2268/4556 [01:06<01:06, 34.30it/s]
50%|####9 | 2272/4556 [01:07<01:07, 33.77it/s]
50%|####9 | 2276/4556 [01:07<01:07, 33.86it/s]
50%|##### | 2280/4556 [01:07<01:06, 33.99it/s]
50%|##### | 2284/4556 [01:07<01:06, 34.14it/s]
50%|##### | 2288/4556 [01:07<01:06, 34.19it/s]
50%|##### | 2292/4556 [01:07<01:06, 34.21it/s]
50%|##### | 2296/4556 [01:07<01:06, 34.19it/s]
50%|##### | 2300/4556 [01:07<01:05, 34.23it/s]
51%|##### | 2304/4556 [01:07<01:05, 34.28it/s]
51%|##### | 2308/4556 [01:08<01:06, 33.81it/s]
51%|##### | 2312/4556 [01:08<01:06, 33.98it/s]
51%|##### | 2316/4556 [01:08<01:05, 34.12it/s]
51%|##### | 2320/4556 [01:08<01:05, 34.23it/s]
51%|#####1 | 2324/4556 [01:08<01:05, 34.24it/s]
51%|#####1 | 2328/4556 [01:08<01:05, 34.21it/s]
51%|#####1 | 2332/4556 [01:08<01:04, 34.27it/s]
51%|#####1 | 2336/4556 [01:08<01:04, 34.30it/s]
51%|#####1 | 2340/4556 [01:08<01:05, 34.01it/s]
51%|#####1 | 2344/4556 [01:09<01:05, 33.84it/s]
52%|#####1 | 2348/4556 [01:09<01:04, 34.04it/s]
52%|#####1 | 2352/4556 [01:09<01:04, 34.12it/s]
52%|#####1 | 2356/4556 [01:09<01:04, 34.20it/s]
52%|#####1 | 2360/4556 [01:09<01:04, 34.29it/s]
52%|#####1 | 2364/4556 [01:09<01:03, 34.30it/s]
52%|#####1 | 2368/4556 [01:09<01:03, 34.30it/s]
52%|#####2 | 2372/4556 [01:09<01:03, 34.29it/s]
52%|#####2 | 2376/4556 [01:10<01:04, 33.97it/s]
52%|#####2 | 2380/4556 [01:10<01:03, 34.13it/s]
52%|#####2 | 2384/4556 [01:10<01:03, 34.22it/s]
52%|#####2 | 2388/4556 [01:10<01:03, 34.22it/s]
53%|#####2 | 2392/4556 [01:10<01:03, 34.30it/s]
53%|#####2 | 2396/4556 [01:10<01:02, 34.29it/s]
53%|#####2 | 2400/4556 [01:10<01:02, 34.27it/s]
53%|#####2 | 2404/4556 [01:10<01:02, 34.20it/s]
53%|#####2 | 2408/4556 [01:10<01:02, 34.32it/s]
53%|#####2 | 2412/4556 [01:11<01:03, 33.76it/s]
53%|#####3 | 2416/4556 [01:11<01:03, 33.91it/s]
53%|#####3 | 2420/4556 [01:11<01:02, 34.11it/s]
53%|#####3 | 2424/4556 [01:11<01:02, 34.25it/s]
53%|#####3 | 2428/4556 [01:11<01:02, 34.31it/s]
53%|#####3 | 2432/4556 [01:11<01:01, 34.31it/s]
53%|#####3 | 2436/4556 [01:11<01:01, 34.30it/s]
54%|#####3 | 2440/4556 [01:11<01:01, 34.28it/s]
54%|#####3 | 2444/4556 [01:12<01:01, 34.16it/s]
54%|#####3 | 2448/4556 [01:12<01:02, 33.62it/s]
54%|#####3 | 2452/4556 [01:12<01:02, 33.63it/s]
54%|#####3 | 2456/4556 [01:12<01:01, 33.89it/s]
54%|#####3 | 2460/4556 [01:12<01:01, 34.01it/s]
54%|#####4 | 2464/4556 [01:12<01:01, 34.04it/s]
54%|#####4 | 2468/4556 [01:12<01:01, 33.99it/s]
54%|#####4 | 2472/4556 [01:12<01:01, 34.10it/s]
54%|#####4 | 2476/4556 [01:12<01:00, 34.15it/s]
54%|#####4 | 2480/4556 [01:13<01:00, 34.11it/s]
55%|#####4 | 2484/4556 [01:13<01:01, 33.64it/s]
55%|#####4 | 2488/4556 [01:13<01:01, 33.71it/s]
55%|#####4 | 2492/4556 [01:13<01:00, 33.88it/s]
55%|#####4 | 2496/4556 [01:13<01:00, 33.98it/s]
55%|#####4 | 2500/4556 [01:13<01:00, 34.03it/s]
55%|#####4 | 2504/4556 [01:13<01:00, 34.08it/s]
55%|#####5 | 2508/4556 [01:13<00:59, 34.20it/s]
55%|#####5 | 2512/4556 [01:14<00:59, 34.19it/s]
55%|#####5 | 2516/4556 [01:14<00:59, 34.24it/s]
55%|#####5 | 2520/4556 [01:14<01:00, 33.71it/s]
55%|#####5 | 2524/4556 [01:14<00:59, 33.87it/s]
55%|#####5 | 2528/4556 [01:14<01:02, 32.63it/s]
56%|#####5 | 2532/4556 [01:14<01:04, 31.32it/s]
56%|#####5 | 2536/4556 [01:14<01:02, 32.13it/s]
56%|#####5 | 2540/4556 [01:14<01:01, 32.77it/s]
56%|#####5 | 2544/4556 [01:15<01:00, 33.18it/s]
56%|#####5 | 2548/4556 [01:15<00:59, 33.54it/s]
56%|#####6 | 2552/4556 [01:15<01:00, 33.26it/s]
56%|#####6 | 2556/4556 [01:15<00:59, 33.51it/s]
56%|#####6 | 2560/4556 [01:15<00:59, 33.70it/s]
56%|#####6 | 2564/4556 [01:15<00:58, 33.94it/s]
56%|#####6 | 2568/4556 [01:15<01:11, 27.89it/s]
56%|#####6 | 2572/4556 [01:15<01:07, 29.55it/s]
57%|#####6 | 2576/4556 [01:16<01:04, 30.82it/s]
57%|#####6 | 2580/4556 [01:16<01:02, 31.82it/s]
57%|#####6 | 2584/4556 [01:16<01:01, 32.15it/s]
57%|#####6 | 2588/4556 [01:16<01:00, 32.76it/s]
57%|#####6 | 2592/4556 [01:16<00:59, 33.22it/s]
57%|#####6 | 2596/4556 [01:16<00:58, 33.53it/s]
57%|#####7 | 2600/4556 [01:16<00:57, 33.78it/s]
57%|#####7 | 2604/4556 [01:16<00:57, 33.95it/s]
57%|#####7 | 2608/4556 [01:16<00:57, 34.09it/s]
57%|#####7 | 2612/4556 [01:17<00:56, 34.13it/s]
57%|#####7 | 2616/4556 [01:17<00:56, 34.22it/s]
58%|#####7 | 2620/4556 [01:17<00:57, 33.73it/s]
58%|#####7 | 2624/4556 [01:17<00:57, 33.85it/s]
58%|#####7 | 2628/4556 [01:17<00:56, 34.00it/s]
58%|#####7 | 2632/4556 [01:17<00:56, 33.99it/s]
58%|#####7 | 2636/4556 [01:17<00:56, 34.05it/s]
58%|#####7 | 2640/4556 [01:17<00:56, 34.04it/s]
58%|#####8 | 2644/4556 [01:18<00:56, 34.14it/s]
58%|#####8 | 2648/4556 [01:18<00:55, 34.19it/s]
58%|#####8 | 2652/4556 [01:18<00:55, 34.20it/s]
58%|#####8 | 2656/4556 [01:18<00:56, 33.70it/s]
58%|#####8 | 2660/4556 [01:18<00:55, 33.96it/s]
58%|#####8 | 2664/4556 [01:18<00:55, 34.07it/s]
59%|#####8 | 2668/4556 [01:18<00:55, 34.09it/s]
59%|#####8 | 2672/4556 [01:18<00:55, 34.15it/s]
59%|#####8 | 2676/4556 [01:18<00:55, 34.16it/s]
59%|#####8 | 2680/4556 [01:19<00:54, 34.22it/s]
59%|#####8 | 2684/4556 [01:19<00:54, 34.26it/s]
59%|#####8 | 2688/4556 [01:19<00:54, 34.25it/s]
59%|#####9 | 2692/4556 [01:19<00:55, 33.85it/s]
59%|#####9 | 2696/4556 [01:19<00:54, 34.00it/s]
59%|#####9 | 2700/4556 [01:19<00:54, 34.21it/s]
59%|#####9 | 2704/4556 [01:19<00:54, 34.24it/s]
59%|#####9 | 2708/4556 [01:19<00:54, 34.22it/s]
60%|#####9 | 2712/4556 [01:20<00:53, 34.16it/s]
60%|#####9 | 2716/4556 [01:20<00:53, 34.16it/s]
60%|#####9 | 2720/4556 [01:20<00:53, 34.23it/s]
60%|#####9 | 2724/4556 [01:20<00:53, 34.21it/s]
60%|#####9 | 2728/4556 [01:20<00:54, 33.79it/s]
60%|#####9 | 2732/4556 [01:20<00:53, 33.96it/s]
60%|###### | 2736/4556 [01:20<00:53, 34.09it/s]
60%|###### | 2740/4556 [01:20<00:53, 34.12it/s]
60%|###### | 2744/4556 [01:20<00:53, 34.18it/s]
60%|###### | 2748/4556 [01:21<00:52, 34.25it/s]
60%|###### | 2752/4556 [01:21<00:52, 34.29it/s]
60%|###### | 2756/4556 [01:21<00:52, 34.35it/s]
61%|###### | 2760/4556 [01:21<00:52, 34.38it/s]
61%|###### | 2764/4556 [01:21<00:52, 33.84it/s]
61%|###### | 2768/4556 [01:21<00:52, 33.92it/s]
61%|###### | 2772/4556 [01:21<00:52, 34.01it/s]
61%|###### | 2776/4556 [01:21<00:52, 34.08it/s]
61%|######1 | 2780/4556 [01:22<00:52, 34.11it/s]
61%|######1 | 2784/4556 [01:22<00:51, 34.21it/s]
61%|######1 | 2788/4556 [01:22<00:51, 34.20it/s]
61%|######1 | 2792/4556 [01:22<00:51, 34.26it/s]
61%|######1 | 2796/4556 [01:22<00:52, 33.77it/s]
61%|######1 | 2800/4556 [01:22<00:51, 33.96it/s]
62%|######1 | 2804/4556 [01:22<00:51, 34.08it/s]
62%|######1 | 2808/4556 [01:22<00:51, 34.13it/s]
62%|######1 | 2812/4556 [01:22<00:51, 34.17it/s]
62%|######1 | 2816/4556 [01:23<00:50, 34.17it/s]
62%|######1 | 2820/4556 [01:23<00:50, 34.13it/s]
62%|######1 | 2824/4556 [01:23<00:50, 34.14it/s]
62%|######2 | 2828/4556 [01:23<00:50, 34.18it/s]
62%|######2 | 2832/4556 [01:23<00:51, 33.68it/s]
62%|######2 | 2836/4556 [01:23<00:50, 33.77it/s]
62%|######2 | 2840/4556 [01:23<00:50, 33.92it/s]
62%|######2 | 2844/4556 [01:23<00:50, 34.09it/s]
63%|######2 | 2848/4556 [01:24<00:50, 34.15it/s]
63%|######2 | 2852/4556 [01:24<00:49, 34.18it/s]
63%|######2 | 2856/4556 [01:24<00:49, 34.18it/s]
63%|######2 | 2860/4556 [01:24<00:49, 34.21it/s]
63%|######2 | 2864/4556 [01:24<00:49, 34.24it/s]
63%|######2 | 2868/4556 [01:24<00:49, 33.78it/s]
63%|######3 | 2872/4556 [01:24<00:49, 33.95it/s]
63%|######3 | 2876/4556 [01:24<00:49, 34.03it/s]
63%|######3 | 2880/4556 [01:24<00:49, 34.03it/s]
63%|######3 | 2884/4556 [01:25<00:49, 34.09it/s]
63%|######3 | 2888/4556 [01:25<00:48, 34.08it/s]
63%|######3 | 2892/4556 [01:25<00:48, 34.16it/s]
64%|######3 | 2896/4556 [01:25<00:48, 34.23it/s]
64%|######3 | 2900/4556 [01:25<00:48, 34.26it/s]
64%|######3 | 2904/4556 [01:25<00:48, 33.77it/s]
64%|######3 | 2908/4556 [01:25<00:48, 33.91it/s]
64%|######3 | 2912/4556 [01:25<00:48, 34.05it/s]
64%|######4 | 2916/4556 [01:26<00:48, 34.14it/s]
64%|######4 | 2920/4556 [01:26<00:47, 34.26it/s]
64%|######4 | 2924/4556 [01:26<00:47, 34.22it/s]
64%|######4 | 2928/4556 [01:26<00:47, 34.28it/s]
64%|######4 | 2932/4556 [01:26<00:47, 34.27it/s]
64%|######4 | 2936/4556 [01:26<00:47, 34.26it/s]
65%|######4 | 2940/4556 [01:26<00:47, 33.84it/s]
65%|######4 | 2944/4556 [01:26<00:47, 34.05it/s]
65%|######4 | 2948/4556 [01:26<00:47, 34.14it/s]
65%|######4 | 2952/4556 [01:27<00:46, 34.18it/s]
65%|######4 | 2956/4556 [01:27<00:46, 34.21it/s]
65%|######4 | 2960/4556 [01:27<00:46, 34.23it/s]
65%|######5 | 2964/4556 [01:27<00:46, 34.23it/s]
65%|######5 | 2968/4556 [01:27<00:46, 34.30it/s]
65%|######5 | 2972/4556 [01:27<00:46, 34.21it/s]
65%|######5 | 2976/4556 [01:27<00:46, 33.76it/s]
65%|######5 | 2980/4556 [01:27<00:46, 33.89it/s]
65%|######5 | 2984/4556 [01:28<00:46, 34.05it/s]
66%|######5 | 2988/4556 [01:28<00:46, 34.08it/s]
66%|######5 | 2992/4556 [01:28<00:45, 34.15it/s]
66%|######5 | 2996/4556 [01:28<00:45, 34.26it/s]
66%|######5 | 3000/4556 [01:28<00:45, 34.23it/s]
66%|######5 | 3004/4556 [01:28<00:45, 34.25it/s]
66%|######6 | 3008/4556 [01:28<00:45, 33.98it/s]
66%|######6 | 3012/4556 [01:28<00:45, 33.85it/s]
66%|######6 | 3016/4556 [01:28<00:45, 33.94it/s]
66%|######6 | 3020/4556 [01:29<00:45, 34.06it/s]
66%|######6 | 3024/4556 [01:29<00:44, 34.13it/s]
66%|######6 | 3028/4556 [01:29<00:44, 34.16it/s]
67%|######6 | 3032/4556 [01:29<00:44, 34.20it/s]
67%|######6 | 3036/4556 [01:29<00:44, 34.21it/s]
67%|######6 | 3040/4556 [01:29<00:44, 34.19it/s]
67%|######6 | 3044/4556 [01:29<00:44, 33.78it/s]
67%|######6 | 3048/4556 [01:29<00:44, 33.95it/s]
67%|######6 | 3052/4556 [01:30<00:44, 34.04it/s]
67%|######7 | 3056/4556 [01:30<00:44, 34.04it/s]
67%|######7 | 3060/4556 [01:30<00:43, 34.07it/s]
67%|######7 | 3064/4556 [01:30<00:43, 34.13it/s]
67%|######7 | 3068/4556 [01:30<00:43, 34.21it/s]
67%|######7 | 3072/4556 [01:30<00:43, 34.33it/s]
68%|######7 | 3076/4556 [01:30<00:43, 34.35it/s]
68%|######7 | 3080/4556 [01:30<00:43, 33.86it/s]
68%|######7 | 3084/4556 [01:30<00:43, 33.95it/s]
68%|######7 | 3088/4556 [01:31<00:43, 33.92it/s]
68%|######7 | 3092/4556 [01:31<00:43, 33.97it/s]
68%|######7 | 3096/4556 [01:31<00:42, 34.08it/s]
68%|######8 | 3100/4556 [01:31<00:42, 34.10it/s]
68%|######8 | 3104/4556 [01:31<00:42, 34.13it/s]
68%|######8 | 3108/4556 [01:31<00:42, 34.11it/s]
68%|######8 | 3112/4556 [01:31<00:42, 34.20it/s]
68%|######8 | 3116/4556 [01:31<00:42, 33.67it/s]
68%|######8 | 3120/4556 [01:32<00:42, 33.80it/s]
69%|######8 | 3124/4556 [01:32<00:42, 33.99it/s]
69%|######8 | 3128/4556 [01:32<00:41, 34.04it/s]
69%|######8 | 3132/4556 [01:32<00:41, 33.98it/s]
69%|######8 | 3136/4556 [01:32<00:41, 34.08it/s]
69%|######8 | 3140/4556 [01:32<00:41, 34.14it/s]
69%|######9 | 3144/4556 [01:32<00:41, 34.17it/s]
69%|######9 | 3148/4556 [01:32<00:41, 34.12it/s]
69%|######9 | 3152/4556 [01:32<00:41, 33.76it/s]
69%|######9 | 3156/4556 [01:33<00:41, 33.82it/s]
69%|######9 | 3160/4556 [01:33<00:41, 33.99it/s]
69%|######9 | 3164/4556 [01:33<00:40, 34.05it/s]
70%|######9 | 3168/4556 [01:33<00:40, 34.12it/s]
70%|######9 | 3172/4556 [01:33<00:40, 34.15it/s]
70%|######9 | 3176/4556 [01:33<00:40, 34.16it/s]
70%|######9 | 3180/4556 [01:33<00:40, 34.22it/s]
70%|######9 | 3184/4556 [01:33<00:40, 34.22it/s]
70%|######9 | 3188/4556 [01:34<00:40, 33.76it/s]
70%|####### | 3192/4556 [01:34<00:40, 33.98it/s]
70%|####### | 3196/4556 [01:34<00:39, 34.07it/s]
70%|####### | 3200/4556 [01:34<00:39, 34.14it/s]
70%|####### | 3204/4556 [01:34<00:39, 34.19it/s]
70%|####### | 3208/4556 [01:34<00:39, 34.23it/s]
71%|####### | 3212/4556 [01:34<00:39, 34.25it/s]
71%|####### | 3216/4556 [01:34<00:39, 34.19it/s]
71%|####### | 3220/4556 [01:34<00:39, 34.10it/s]
71%|####### | 3224/4556 [01:35<00:39, 33.78it/s]
71%|####### | 3228/4556 [01:35<00:39, 33.81it/s]
71%|####### | 3232/4556 [01:35<00:38, 33.96it/s]
71%|#######1 | 3236/4556 [01:35<00:38, 34.03it/s]
71%|#######1 | 3240/4556 [01:35<00:38, 34.09it/s]
71%|#######1 | 3244/4556 [01:35<00:38, 34.16it/s]
71%|#######1 | 3248/4556 [01:35<00:38, 34.17it/s]
71%|#######1 | 3252/4556 [01:35<00:38, 34.18it/s]
71%|#######1 | 3256/4556 [01:35<00:38, 33.80it/s]
72%|#######1 | 3260/4556 [01:36<00:38, 33.93it/s]
72%|#######1 | 3264/4556 [01:36<00:37, 34.08it/s]
72%|#######1 | 3268/4556 [01:36<00:37, 34.19it/s]
72%|#######1 | 3272/4556 [01:36<00:37, 34.18it/s]
72%|#######1 | 3276/4556 [01:36<00:37, 34.13it/s]
72%|#######1 | 3280/4556 [01:36<00:37, 34.19it/s]
72%|#######2 | 3284/4556 [01:36<00:37, 34.16it/s]
72%|#######2 | 3288/4556 [01:36<00:37, 34.13it/s]
72%|#######2 | 3292/4556 [01:37<00:37, 33.73it/s]
72%|#######2 | 3296/4556 [01:37<00:37, 33.94it/s]
72%|#######2 | 3300/4556 [01:37<00:36, 34.04it/s]
73%|#######2 | 3304/4556 [01:37<00:36, 34.10it/s]
73%|#######2 | 3308/4556 [01:37<00:36, 34.04it/s]
73%|#######2 | 3312/4556 [01:37<00:36, 34.10it/s]
73%|#######2 | 3316/4556 [01:37<00:36, 34.24it/s]
73%|#######2 | 3320/4556 [01:37<00:36, 34.25it/s]
73%|#######2 | 3324/4556 [01:38<00:44, 27.80it/s]
73%|#######3 | 3328/4556 [01:38<00:41, 29.46it/s]
73%|#######3 | 3332/4556 [01:38<00:39, 30.73it/s]
73%|#######3 | 3336/4556 [01:38<00:38, 31.70it/s]
73%|#######3 | 3340/4556 [01:38<00:37, 32.39it/s]
73%|#######3 | 3344/4556 [01:38<00:36, 32.92it/s]
73%|#######3 | 3348/4556 [01:38<00:36, 33.29it/s]
74%|#######3 | 3352/4556 [01:38<00:35, 33.56it/s]
74%|#######3 | 3356/4556 [01:39<00:35, 33.80it/s]
74%|#######3 | 3360/4556 [01:39<00:35, 33.55it/s]
74%|#######3 | 3364/4556 [01:39<00:35, 33.80it/s]
74%|#######3 | 3368/4556 [01:39<00:35, 33.94it/s]
74%|#######4 | 3372/4556 [01:39<00:34, 34.08it/s]
74%|#######4 | 3376/4556 [01:39<00:34, 34.06it/s]
74%|#######4 | 3380/4556 [01:39<00:34, 34.13it/s]
74%|#######4 | 3384/4556 [01:39<00:34, 34.17it/s]
74%|#######4 | 3388/4556 [01:39<00:34, 34.14it/s]
74%|#######4 | 3392/4556 [01:40<00:34, 34.12it/s]
75%|#######4 | 3396/4556 [01:40<00:34, 33.71it/s]
75%|#######4 | 3400/4556 [01:40<00:34, 33.84it/s]
75%|#######4 | 3404/4556 [01:40<00:33, 34.00it/s]
75%|#######4 | 3408/4556 [01:40<00:33, 34.08it/s]
75%|#######4 | 3412/4556 [01:40<00:33, 34.13it/s]
75%|#######4 | 3416/4556 [01:40<00:33, 34.17it/s]
75%|#######5 | 3420/4556 [01:40<00:35, 31.70it/s]
75%|#######5 | 3424/4556 [01:41<00:34, 32.46it/s]
75%|#######5 | 3428/4556 [01:41<00:34, 32.72it/s]
75%|#######5 | 3432/4556 [01:41<00:34, 32.86it/s]
75%|#######5 | 3436/4556 [01:41<00:33, 33.25it/s]
76%|#######5 | 3440/4556 [01:41<00:33, 33.63it/s]
76%|#######5 | 3444/4556 [01:41<00:32, 33.80it/s]
76%|#######5 | 3448/4556 [01:41<00:32, 34.01it/s]
76%|#######5 | 3452/4556 [01:41<00:32, 34.09it/s]
76%|#######5 | 3456/4556 [01:41<00:32, 34.17it/s]
76%|#######5 | 3460/4556 [01:42<00:32, 34.25it/s]
76%|#######6 | 3464/4556 [01:42<00:32, 33.82it/s]
76%|#######6 | 3468/4556 [01:42<00:32, 33.98it/s]
76%|#######6 | 3472/4556 [01:42<00:31, 34.05it/s]
76%|#######6 | 3476/4556 [01:42<00:31, 34.14it/s]
76%|#######6 | 3480/4556 [01:42<00:31, 34.23it/s]
76%|#######6 | 3484/4556 [01:42<00:31, 34.24it/s]
77%|#######6 | 3488/4556 [01:42<00:31, 34.21it/s]
77%|#######6 | 3492/4556 [01:43<00:31, 34.23it/s]
77%|#######6 | 3496/4556 [01:43<00:30, 34.27it/s]
77%|#######6 | 3500/4556 [01:43<00:31, 33.78it/s]
77%|#######6 | 3504/4556 [01:43<00:31, 33.92it/s]
77%|#######6 | 3508/4556 [01:43<00:30, 34.07it/s]
77%|#######7 | 3512/4556 [01:43<00:30, 34.16it/s]
77%|#######7 | 3516/4556 [01:43<00:30, 34.24it/s]
77%|#######7 | 3520/4556 [01:43<00:30, 34.24it/s]
77%|#######7 | 3524/4556 [01:43<00:30, 34.27it/s]
77%|#######7 | 3528/4556 [01:44<00:29, 34.30it/s]
78%|#######7 | 3532/4556 [01:44<00:29, 34.31it/s]
78%|#######7 | 3536/4556 [01:44<00:30, 33.89it/s]
78%|#######7 | 3540/4556 [01:44<00:29, 33.98it/s]
78%|#######7 | 3544/4556 [01:44<00:29, 34.07it/s]
78%|#######7 | 3548/4556 [01:44<00:29, 34.13it/s]
78%|#######7 | 3552/4556 [01:44<00:29, 34.19it/s]
78%|#######8 | 3556/4556 [01:44<00:29, 34.25it/s]
78%|#######8 | 3560/4556 [01:45<00:29, 34.17it/s]
78%|#######8 | 3564/4556 [01:45<00:29, 34.18it/s]
78%|#######8 | 3568/4556 [01:45<00:28, 34.19it/s]
78%|#######8 | 3572/4556 [01:45<00:29, 33.75it/s]
78%|#######8 | 3576/4556 [01:45<00:29, 33.79it/s]
79%|#######8 | 3580/4556 [01:45<00:28, 33.95it/s]
79%|#######8 | 3584/4556 [01:45<00:28, 34.09it/s]
79%|#######8 | 3588/4556 [01:45<00:28, 34.02it/s]
79%|#######8 | 3592/4556 [01:45<00:28, 34.17it/s]
79%|#######8 | 3596/4556 [01:46<00:28, 34.12it/s]
79%|#######9 | 3600/4556 [01:46<00:28, 34.08it/s]
79%|#######9 | 3604/4556 [01:46<00:27, 34.14it/s]
79%|#######9 | 3608/4556 [01:46<00:28, 33.62it/s]
79%|#######9 | 3612/4556 [01:46<00:27, 33.83it/s]
79%|#######9 | 3616/4556 [01:46<00:28, 33.49it/s]
79%|#######9 | 3620/4556 [01:46<00:28, 33.19it/s]
80%|#######9 | 3624/4556 [01:46<00:27, 33.44it/s]
80%|#######9 | 3628/4556 [01:47<00:27, 33.68it/s]
80%|#######9 | 3632/4556 [01:47<00:27, 33.85it/s]
80%|#######9 | 3636/4556 [01:47<00:27, 33.98it/s]
80%|#######9 | 3640/4556 [01:47<00:27, 32.97it/s]
80%|#######9 | 3644/4556 [01:47<00:28, 32.50it/s]
80%|######## | 3648/4556 [01:47<00:27, 32.96it/s]
80%|######## | 3652/4556 [01:47<00:27, 33.32it/s]
80%|######## | 3656/4556 [01:47<00:26, 33.66it/s]
80%|######## | 3660/4556 [01:47<00:26, 33.87it/s]
80%|######## | 3664/4556 [01:48<00:26, 34.04it/s]
81%|######## | 3668/4556 [01:48<00:26, 34.09it/s]
81%|######## | 3672/4556 [01:48<00:25, 34.25it/s]
81%|######## | 3676/4556 [01:48<00:25, 34.26it/s]
81%|######## | 3680/4556 [01:48<00:25, 33.82it/s]
81%|######## | 3684/4556 [01:48<00:25, 33.93it/s]
81%|######## | 3688/4556 [01:48<00:25, 34.01it/s]
81%|########1 | 3692/4556 [01:48<00:25, 34.10it/s]
81%|########1 | 3696/4556 [01:49<00:25, 34.15it/s]
81%|########1 | 3700/4556 [01:49<00:25, 34.20it/s]
81%|########1 | 3704/4556 [01:49<00:24, 34.19it/s]
81%|########1 | 3708/4556 [01:49<00:24, 34.24it/s]
81%|########1 | 3712/4556 [01:49<00:24, 34.16it/s]
82%|########1 | 3716/4556 [01:49<00:24, 33.74it/s]
82%|########1 | 3720/4556 [01:49<00:24, 33.82it/s]
82%|########1 | 3724/4556 [01:49<00:24, 33.94it/s]
82%|########1 | 3728/4556 [01:49<00:24, 33.97it/s]
82%|########1 | 3732/4556 [01:50<00:24, 33.99it/s]
82%|########2 | 3736/4556 [01:50<00:24, 34.01it/s]
82%|########2 | 3740/4556 [01:50<00:23, 34.11it/s]
82%|########2 | 3744/4556 [01:50<00:23, 34.17it/s]
82%|########2 | 3748/4556 [01:50<00:23, 33.80it/s]
82%|########2 | 3752/4556 [01:50<00:23, 33.91it/s]
82%|########2 | 3756/4556 [01:50<00:23, 34.02it/s]
83%|########2 | 3760/4556 [01:50<00:23, 34.11it/s]
83%|########2 | 3764/4556 [01:51<00:23, 34.12it/s]
83%|########2 | 3768/4556 [01:51<00:23, 34.18it/s]
83%|########2 | 3772/4556 [01:51<00:22, 34.15it/s]
83%|########2 | 3776/4556 [01:51<00:22, 34.21it/s]
83%|########2 | 3780/4556 [01:51<00:22, 34.26it/s]
83%|########3 | 3784/4556 [01:51<00:22, 33.76it/s]
83%|########3 | 3788/4556 [01:51<00:22, 33.94it/s]
83%|########3 | 3792/4556 [01:51<00:22, 33.96it/s]
83%|########3 | 3796/4556 [01:51<00:22, 34.05it/s]
83%|########3 | 3800/4556 [01:52<00:22, 34.12it/s]
83%|########3 | 3804/4556 [01:52<00:21, 34.19it/s]
84%|########3 | 3808/4556 [01:52<00:21, 34.16it/s]
84%|########3 | 3812/4556 [01:52<00:21, 34.07it/s]
84%|########3 | 3816/4556 [01:52<00:21, 34.16it/s]
84%|########3 | 3820/4556 [01:52<00:21, 33.75it/s]
84%|########3 | 3824/4556 [01:52<00:21, 33.91it/s]
84%|########4 | 3828/4556 [01:52<00:21, 33.99it/s]
84%|########4 | 3832/4556 [01:53<00:21, 34.09it/s]
84%|########4 | 3836/4556 [01:53<00:21, 34.13it/s]
84%|########4 | 3840/4556 [01:53<00:20, 34.18it/s]
84%|########4 | 3844/4556 [01:53<00:20, 34.13it/s]
84%|########4 | 3848/4556 [01:53<00:20, 34.17it/s]
85%|########4 | 3852/4556 [01:53<00:20, 34.25it/s]
85%|########4 | 3856/4556 [01:53<00:20, 33.78it/s]
85%|########4 | 3860/4556 [01:53<00:20, 33.96it/s]
85%|########4 | 3864/4556 [01:53<00:20, 34.04it/s]
85%|########4 | 3868/4556 [01:54<00:20, 34.10it/s]
85%|########4 | 3872/4556 [01:54<00:20, 34.18it/s]
85%|########5 | 3876/4556 [01:54<00:19, 34.24it/s]
85%|########5 | 3880/4556 [01:54<00:19, 34.20it/s]
85%|########5 | 3884/4556 [01:54<00:19, 34.22it/s]
85%|########5 | 3888/4556 [01:54<00:19, 34.30it/s]
85%|########5 | 3892/4556 [01:54<00:19, 33.74it/s]
86%|########5 | 3896/4556 [01:54<00:19, 33.91it/s]
86%|########5 | 3900/4556 [01:55<00:19, 33.99it/s]
86%|########5 | 3904/4556 [01:55<00:19, 34.04it/s]
86%|########5 | 3908/4556 [01:55<00:18, 34.14it/s]
86%|########5 | 3912/4556 [01:55<00:18, 34.17it/s]
86%|########5 | 3916/4556 [01:55<00:18, 34.21it/s]
86%|########6 | 3920/4556 [01:55<00:18, 34.24it/s]
86%|########6 | 3924/4556 [01:55<00:18, 34.18it/s]
86%|########6 | 3928/4556 [01:55<00:18, 33.78it/s]
86%|########6 | 3932/4556 [01:55<00:18, 33.91it/s]
86%|########6 | 3936/4556 [01:56<00:18, 33.98it/s]
86%|########6 | 3940/4556 [01:56<00:18, 34.01it/s]
87%|########6 | 3944/4556 [01:56<00:17, 34.03it/s]
87%|########6 | 3948/4556 [01:56<00:17, 34.18it/s]
87%|########6 | 3952/4556 [01:56<00:17, 34.17it/s]
87%|########6 | 3956/4556 [01:56<00:17, 34.17it/s]
87%|########6 | 3960/4556 [01:56<00:17, 33.79it/s]
87%|########7 | 3964/4556 [01:56<00:17, 33.92it/s]
87%|########7 | 3968/4556 [01:57<00:17, 34.05it/s]
87%|########7 | 3972/4556 [01:57<00:17, 34.13it/s]
87%|########7 | 3976/4556 [01:57<00:17, 34.09it/s]
87%|########7 | 3980/4556 [01:57<00:16, 34.13it/s]
87%|########7 | 3984/4556 [01:57<00:16, 34.23it/s]
88%|########7 | 3988/4556 [01:57<00:16, 34.13it/s]
88%|########7 | 3992/4556 [01:57<00:16, 34.18it/s]
88%|########7 | 3996/4556 [01:57<00:16, 33.64it/s]
88%|########7 | 4000/4556 [01:57<00:16, 33.81it/s]
88%|########7 | 4004/4556 [01:58<00:16, 33.85it/s]
88%|########7 | 4008/4556 [01:58<00:16, 33.96it/s]
88%|########8 | 4012/4556 [01:58<00:15, 34.07it/s]
88%|########8 | 4016/4556 [01:58<00:15, 34.05it/s]
88%|########8 | 4020/4556 [01:58<00:15, 34.12it/s]
88%|########8 | 4024/4556 [01:58<00:15, 34.17it/s]
88%|########8 | 4028/4556 [01:58<00:15, 34.18it/s]
88%|########8 | 4032/4556 [01:58<00:15, 33.70it/s]
89%|########8 | 4036/4556 [01:59<00:15, 33.76it/s]
89%|########8 | 4040/4556 [01:59<00:15, 33.92it/s]
89%|########8 | 4044/4556 [01:59<00:15, 34.04it/s]
89%|########8 | 4048/4556 [01:59<00:14, 34.13it/s]
89%|########8 | 4052/4556 [01:59<00:14, 34.15it/s]
89%|########9 | 4056/4556 [01:59<00:14, 34.16it/s]
89%|########9 | 4060/4556 [01:59<00:14, 34.18it/s]
89%|########9 | 4064/4556 [01:59<00:14, 34.11it/s]
89%|########9 | 4068/4556 [01:59<00:14, 33.73it/s]
89%|########9 | 4072/4556 [02:00<00:14, 33.86it/s]
89%|########9 | 4076/4556 [02:00<00:14, 33.96it/s]
90%|########9 | 4080/4556 [02:00<00:17, 27.81it/s]
90%|########9 | 4084/4556 [02:00<00:16, 29.48it/s]
90%|########9 | 4088/4556 [02:00<00:15, 30.72it/s]
90%|########9 | 4092/4556 [02:00<00:14, 31.71it/s]
90%|########9 | 4096/4556 [02:00<00:14, 32.41it/s]
90%|########9 | 4100/4556 [02:01<00:14, 32.39it/s]
90%|######### | 4104/4556 [02:01<00:13, 32.90it/s]
90%|######### | 4108/4556 [02:01<00:13, 33.28it/s]
90%|######### | 4112/4556 [02:01<00:13, 33.52it/s]
90%|######### | 4116/4556 [02:01<00:13, 33.72it/s]
90%|######### | 4120/4556 [02:01<00:12, 33.84it/s]
91%|######### | 4124/4556 [02:01<00:12, 33.76it/s]
91%|######### | 4128/4556 [02:01<00:12, 33.81it/s]
91%|######### | 4132/4556 [02:01<00:12, 33.93it/s]
91%|######### | 4136/4556 [02:02<00:12, 33.47it/s]
91%|######### | 4140/4556 [02:02<00:12, 33.60it/s]
91%|######### | 4144/4556 [02:02<00:12, 33.74it/s]
91%|#########1| 4148/4556 [02:02<00:12, 33.87it/s]
91%|#########1| 4152/4556 [02:02<00:11, 33.98it/s]
91%|#########1| 4156/4556 [02:02<00:11, 34.03it/s]
91%|#########1| 4160/4556 [02:02<00:11, 34.09it/s]
91%|#########1| 4164/4556 [02:02<00:11, 34.12it/s]
91%|#########1| 4168/4556 [02:03<00:11, 34.16it/s]
92%|#########1| 4172/4556 [02:03<00:11, 33.68it/s]
92%|#########1| 4176/4556 [02:03<00:11, 33.74it/s]
92%|#########1| 4180/4556 [02:03<00:11, 33.86it/s]
92%|#########1| 4184/4556 [02:03<00:10, 33.95it/s]
92%|#########1| 4188/4556 [02:03<00:10, 33.96it/s]
92%|#########2| 4192/4556 [02:03<00:10, 34.02it/s]
92%|#########2| 4196/4556 [02:03<00:10, 34.06it/s]
92%|#########2| 4200/4556 [02:03<00:10, 34.09it/s]
92%|#########2| 4204/4556 [02:04<00:10, 33.70it/s]
92%|#########2| 4208/4556 [02:04<00:10, 33.78it/s]
92%|#########2| 4212/4556 [02:04<00:10, 33.93it/s]
93%|#########2| 4216/4556 [02:04<00:09, 34.01it/s]
93%|#########2| 4220/4556 [02:04<00:09, 34.09it/s]
93%|#########2| 4224/4556 [02:04<00:09, 34.09it/s]
93%|#########2| 4228/4556 [02:04<00:09, 34.12it/s]
93%|#########2| 4232/4556 [02:04<00:09, 34.08it/s]
93%|#########2| 4236/4556 [02:05<00:09, 34.12it/s]
93%|#########3| 4240/4556 [02:05<00:09, 33.58it/s]
93%|#########3| 4244/4556 [02:05<00:09, 33.78it/s]
93%|#########3| 4248/4556 [02:05<00:09, 33.92it/s]
93%|#########3| 4252/4556 [02:05<00:08, 33.97it/s]
93%|#########3| 4256/4556 [02:05<00:08, 34.02it/s]
94%|#########3| 4260/4556 [02:05<00:08, 34.05it/s]
94%|#########3| 4264/4556 [02:05<00:08, 34.08it/s]
94%|#########3| 4268/4556 [02:05<00:08, 34.11it/s]
94%|#########3| 4272/4556 [02:06<00:08, 34.05it/s]
94%|#########3| 4276/4556 [02:06<00:08, 33.55it/s]
94%|#########3| 4280/4556 [02:06<00:08, 33.69it/s]
94%|#########4| 4284/4556 [02:06<00:08, 33.81it/s]
94%|#########4| 4288/4556 [02:06<00:07, 33.89it/s]
94%|#########4| 4292/4556 [02:06<00:07, 34.00it/s]
94%|#########4| 4296/4556 [02:06<00:07, 34.00it/s]
94%|#########4| 4300/4556 [02:06<00:07, 33.81it/s]
94%|#########4| 4304/4556 [02:07<00:07, 33.89it/s]
95%|#########4| 4308/4556 [02:07<00:07, 33.97it/s]
95%|#########4| 4312/4556 [02:07<00:07, 33.62it/s]
95%|#########4| 4316/4556 [02:07<00:07, 33.80it/s]
95%|#########4| 4320/4556 [02:07<00:06, 33.95it/s]
95%|#########4| 4324/4556 [02:07<00:06, 33.95it/s]
95%|#########4| 4328/4556 [02:07<00:06, 33.99it/s]
95%|#########5| 4332/4556 [02:07<00:06, 34.05it/s]
95%|#########5| 4336/4556 [02:07<00:06, 34.05it/s]
95%|#########5| 4340/4556 [02:08<00:06, 34.08it/s]
95%|#########5| 4344/4556 [02:08<00:06, 34.16it/s]
95%|#########5| 4348/4556 [02:08<00:06, 33.64it/s]
96%|#########5| 4352/4556 [02:08<00:06, 33.80it/s]
96%|#########5| 4356/4556 [02:08<00:05, 33.89it/s]
96%|#########5| 4360/4556 [02:08<00:05, 33.97it/s]
96%|#########5| 4364/4556 [02:08<00:05, 33.99it/s]
96%|#########5| 4368/4556 [02:08<00:05, 34.01it/s]
96%|#########5| 4372/4556 [02:09<00:05, 34.04it/s]
96%|#########6| 4376/4556 [02:09<00:05, 34.06it/s]
96%|#########6| 4380/4556 [02:09<00:05, 33.73it/s]
96%|#########6| 4384/4556 [02:09<00:05, 33.62it/s]
96%|#########6| 4388/4556 [02:09<00:04, 33.76it/s]
96%|#########6| 4392/4556 [02:09<00:04, 33.95it/s]
96%|#########6| 4396/4556 [02:09<00:04, 33.99it/s]
97%|#########6| 4400/4556 [02:09<00:04, 34.08it/s]
97%|#########6| 4404/4556 [02:09<00:04, 34.10it/s]
97%|#########6| 4408/4556 [02:10<00:04, 34.07it/s]
97%|#########6| 4412/4556 [02:10<00:04, 34.06it/s]
97%|#########6| 4416/4556 [02:10<00:04, 33.61it/s]
97%|#########7| 4420/4556 [02:10<00:04, 33.77it/s]
97%|#########7| 4424/4556 [02:10<00:03, 33.84it/s]
97%|#########7| 4428/4556 [02:10<00:03, 33.91it/s]
97%|#########7| 4432/4556 [02:10<00:03, 33.97it/s]
97%|#########7| 4436/4556 [02:10<00:03, 33.94it/s]
97%|#########7| 4440/4556 [02:11<00:03, 34.01it/s]
98%|#########7| 4444/4556 [02:11<00:03, 34.05it/s]
98%|#########7| 4448/4556 [02:11<00:03, 34.07it/s]
98%|#########7| 4452/4556 [02:11<00:03, 33.62it/s]
98%|#########7| 4456/4556 [02:11<00:02, 33.78it/s]
98%|#########7| 4460/4556 [02:11<00:02, 33.90it/s]
98%|#########7| 4464/4556 [02:11<00:02, 33.95it/s]
98%|#########8| 4468/4556 [02:11<00:02, 33.99it/s]
98%|#########8| 4472/4556 [02:11<00:02, 34.05it/s]
98%|#########8| 4476/4556 [02:12<00:02, 34.06it/s]
98%|#########8| 4480/4556 [02:12<00:02, 34.00it/s]
98%|#########8| 4484/4556 [02:12<00:02, 34.03it/s]
99%|#########8| 4488/4556 [02:12<00:02, 33.59it/s]
99%|#########8| 4492/4556 [02:12<00:01, 33.60it/s]
99%|#########8| 4496/4556 [02:12<00:01, 33.74it/s]
99%|#########8| 4500/4556 [02:12<00:01, 33.77it/s]
99%|#########8| 4504/4556 [02:12<00:01, 33.75it/s]
99%|#########8| 4508/4556 [02:13<00:01, 33.86it/s]
99%|#########9| 4512/4556 [02:13<00:01, 33.88it/s]
99%|#########9| 4516/4556 [02:13<00:01, 33.94it/s]
99%|#########9| 4520/4556 [02:13<00:01, 33.98it/s]
99%|#########9| 4524/4556 [02:13<00:00, 32.51it/s]
99%|#########9| 4528/4556 [02:13<00:00, 32.93it/s]
99%|#########9| 4532/4556 [02:13<00:00, 33.24it/s]
100%|#########9| 4536/4556 [02:13<00:00, 33.45it/s]
100%|#########9| 4540/4556 [02:13<00:00, 33.65it/s]
100%|#########9| 4544/4556 [02:14<00:00, 33.77it/s]
100%|#########9| 4548/4556 [02:14<00:00, 33.84it/s]
100%|#########9| 4552/4556 [02:14<00:00, 33.92it/s]
100%|##########| 4556/4556 [02:14<00:00, 31.25it/s]
Link Prediction AUC: 0.9961242858420715
Conclusion¶
In this tutorial, you have learned how to train a multi-layer GraphSAGE for link prediction with neighbor sampling.
# Thumbnail credits: Link Prediction with Neo4j, Mark Needham
# sphinx_gallery_thumbnail_path = '_static/blitz_4_link_predict.png'
Total running time of the script: ( 4 minutes 0.792 seconds)