Batch Size #246
-
|
The FTC-ML Manual lists that the batch size is 32, but the popup shows 512. We have a training set with 5,965 images. We ran it with the formula in the manual but used the 512 batch size and got 1,200 steps. However, we noted that the images on the model only included 2 of our labels. When we decided to use the batch size of 32 in the formula we come up with that we need 19K steps, but it will only allow us to go up to 4K steps on the modal box. Just looking to understand which one is correct and if the batch size of 32 is correct and we should run 19K steps, how would we do that with the 4K limit? Do we just use "More Training" at 4K intervals to add more steps to the model and it starts where left off? I think that might be what we are reading in the manual? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
The manual is specifically describing ftc-ml, which is using GPU training; for GPU, the batch size for most models is 32. If you're using fmltc, you have the option of using TPU or GPU. The TPU batch size for many models is 512 (TPU has more memory and can train more images per step). "More Training" does indeed simply "pick up where it left off" (at least where it left off from that saved checkpoint). If you want to train a model longer, this is the way to do it. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for that clarification that helps a lot. Is the fact that we only see 10 pages of images and doesn't include images of all the labels, a reflection of a system limitation then and not an indication that not all labels were trained and evaluated? |
Beta Was this translation helpful? Give feedback.
-
|
How many evaluation images are in the dataset(s) that you used? All of the evaluation images will be used for evaluation, however we limit the number of visualizations to 100. |
Beta Was this translation helpful? Give feedback.
-
|
Definitely more than 100. I think that explains it. Thank you. |
Beta Was this translation helpful? Give feedback.
The manual is specifically describing ftc-ml, which is using GPU training; for GPU, the batch size for most models is 32. If you're using fmltc, you have the option of using TPU or GPU. The TPU batch size for many models is 512 (TPU has more memory and can train more images per step).
"More Training" does indeed simply "pick up where it left off" (at least where it left off from that saved checkpoint). If you want to train a model longer, this is the way to do it.