2021-04-07 12:20:34.004650: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0

2021-04-07 12:20:35.172713: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set

2021-04-07 12:20:35.173328: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1

2021-04-07 12:20:35.205268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:

pciBusID: 0000:65:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5

coreClock: 1.77GHz coreCount: 40 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s

2021-04-07 12:20:35.205308: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0

2021-04-07 12:20:35.207212: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11

2021-04-07 12:20:35.207269: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11

2021-04-07 12:20:35.207901: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10

2021-04-07 12:20:35.208078: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10

2021-04-07 12:20:35.209716: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10

2021-04-07 12:20:35.210114: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11

2021-04-07 12:20:35.210195: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8

2021-04-07 12:20:35.210887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0

2021-04-07 12:20:35.211149: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX512F

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2021-04-07 12:20:35.211835: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set

2021-04-07 12:20:35.212281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:

pciBusID: 0000:65:00.0 name: GeForce RTX 2070 SUPER computeCapability: 7.5

coreClock: 1.77GHz coreCount: 40 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s

2021-04-07 12:20:35.212302: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0

2021-04-07 12:20:35.212327: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11

2021-04-07 12:20:35.212336: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11

2021-04-07 12:20:35.212344: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10

2021-04-07 12:20:35.212353: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10

2021-04-07 12:20:35.212361: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10

2021-04-07 12:20:35.212368: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11

2021-04-07 12:20:35.212377: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8

2021-04-07 12:20:35.213055: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0

2021-04-07 12:20:35.213075: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0

2021-04-07 12:20:35.587321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:

2021-04-07 12:20:35.587357: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0

2021-04-07 12:20:35.587363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N

2021-04-07 12:20:35.588383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6296 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2070 SUPER, pci bus id: 0000:65:00.0, compute capability: 7.5)

/usr/lib/python3/dist-packages/Crypto/Random/Fortuna/FortunaGenerator.py:28: SyntaxWarning: "is" with a literal. Did you mean "=="?

if sys.version_info[0] is 2 and sys.version_info[1] is 1:

/usr/lib/python3/dist-packages/Crypto/Random/Fortuna/FortunaGenerator.py:28: SyntaxWarning: "is" with a literal. Did you mean "=="?

if sys.version_info[0] is 2 and sys.version_info[1] is 1:

I0407 12:20:36.378304 139902763542336 train_utils.py:285] Final experiment parameters: {'runtime': {'all_reduce_alg': None,

'batchnorm_spatial_persistent': False,

'dataset_num_private_threads': None,

'default_shard_dim': -1,

'distribution_strategy': 'mirrored',

'enable_xla': False,

'gpu_thread_mode': None,

'loss_scale': 'dynamic',

'mixed_precision_dtype': 'float16',

'num_cores_per_replica': 1,

'num_gpus': 1,

'num_packs': 1,

'per_gpu_thread_count': 0,

'run_eagerly': False,

'task_index': -1,

'tpu': None,

'worker_hosts': None},

'task': {'annotation_file': None,

'darknet_load_decoder': False,

'gradient_clip_norm': 0.0,

'init_checkpoint': 'gs://tensorflow2/darknet/cspdarknet53-golden',

'init_checkpoint_modules': 'backbone',

'load_darknet_weights': False,

'model': {'_boxes': ['[12.0, 16.0]',

'[19.0, 36.0]',

'[40.0, 28.0]',

'[36.0, 75.0]',

'[76.0, 55.0]',

'[72.0, 146.0]',

'[142.0, 110.0]',

'[192.0, 243.0]',

'[459.0, 401.0]'],

'_input_size': None,

'base': {'type': None},

'boxes_per_scale': 3,

'decoder_activation': 'leaky',

'filter': {'anchor_generation_scale': 512,

'cls_normalizer': {'3': 0.5, '4': 0.5, '5': 0.5},

'ignore_thresh': {'3': 0.7, '4': 0.7, '5': 0.7},

'iou_normalizer': {'3': 0.05,

'4': 0.05,

'5': 0.05},

'iou_thresh': 0.001,

'loss_type': {'3': 'ciou',

'4': 'ciou',

'5': 'ciou'},

'max_boxes': 200,

'max_delta': {'3': inf, '4': inf, '5': inf},

'max_level': 5,

'min_level': 3,

'new_cords': {'3': True, '4': True, '5': True},

'nms_thresh': 0.6,

'obj_normalizer': {'3': 4.0, '4': 1.0, '5': 0.4},

'objectness_smooth': {'3': 0.0,

'4': 0.0,

'5': 0.0},

'path_scales': {'3': 8, '4': 16, '5': 32},

'scale_xy': {'3': 2.0, '4': 2.0, '5': 2.0},

'truth_thresh': {'3': 1.0, '4': 1.0, '5': 1.0},

'use_nms': True,

'use_reduction_sum': False},

'max_level': 5,

'min_level': 3,

'norm_activation': {'activation': 'mish',

'norm_epsilon': 0.001,

'norm_momentum': 0.99,

'use_sync_bn': True},

'num_classes': 91,

'subdivisions': 1,

'use_sam': False},

'per_category_metrics': False,

'train_data': {'block_length': 1,

'cache': False,

'cycle_length': None,

'decoder': {'simple_decoder': {'regenerate_source_id': False},

'type': 'simple_decoder'},

'deterministic': None,

'drop_remainder': True,

'dtype': 'float16',

'enable_tf_data_service': False,

'global_batch_size': 4,

'input_path': 'gs://tensorflow2/coco_records/train/2017*',

'is_training': True,

'parser': {'aug_rand_angle': 0.0,

'aug_rand_brightness': 1.5,

'aug_rand_hue': 0.1,

'aug_rand_saturation': 1.5,

'aug_rand_translate': 0.2,

'aug_rand_zoom': 0.5,

'fixed_size': True,

'image_h': 512,

'image_w': 512,

'jitter_boxes': 0.4,

'jitter_im': 0.3,

'letter_box': False,

'max_num_instances': 200,

'min_process_size': 512,

'mosaic': {'crop_area': [0.25, 1.0],

'crop_area_mosaic': [0.25, 0.75],

'mosaic_frequency': 0.85,

'output_size': [640, 640],

'random_crop': True,

'random_crop_mosaic': False},

'pct_rand': 0.0,

'random_flip': True,

'use_scale_xy': False,

'use_tie_breaker': True},

'sharding': True,

'shuffle_buffer_size': 10000,

'tf_data_service_address': None,

'tf_data_service_job_name': None,

'tfds_as_supervised': False,

'tfds_data_dir': '',

'tfds_download': True,

'tfds_name': None,

'tfds_skip_decoding_feature': '',

'tfds_split': None},

'validation_data': {'block_length': 1,

'cache': False,

'cycle_length': None,

'decoder': {'simple_decoder': {'regenerate_source_id': False},

'type': 'simple_decoder'},

'deterministic': None,

'drop_remainder': True,

'dtype': 'float16',

'enable_tf_data_service': False,

'global_batch_size': 8,

'input_path': 'gs://tensorflow2/coco_records/val/2017*',

'is_training': False,

'parser': {'aug_rand_angle': 0.0,

'aug_rand_brightness': 1.5,

'aug_rand_hue': 0.1,

'aug_rand_saturation': 1.5,

'aug_rand_translate': 0.35,

'aug_rand_zoom': 0.5,

'fixed_size': True,

'image_h': 608,

'image_w': 608,

'jitter_boxes': 0.6,

'jitter_im': 0.3,

'letter_box': False,

'max_num_instances': 200,

'min_process_size': 320,

'mosaic': {'crop_area': [0.25, 1.0],

'crop_area_mosaic': [0.25,

0.75],

'mosaic_frequency': 0.85,

'output_size': [640, 640],

'random_crop': False,

'random_crop_mosaic': False},

'pct_rand': 0.0,

'random_flip': True,

'use_scale_xy': True,

'use_tie_breaker': True},

'sharding': True,

'shuffle_buffer_size': 10000,

'tf_data_service_address': None,

'tf_data_service_job_name': None,

'tfds_as_supervised': False,

'tfds_data_dir': '',

'tfds_download': True,

'tfds_name': None,

'tfds_skip_decoding_feature': '',

'tfds_split': None},

'weight_decay': 0.001},

'trainer': {'allow_tpu_summary': False,

'best_checkpoint_eval_metric': 'str = "AP50"',

'best_checkpoint_export_subdir': '',

'best_checkpoint_metric_comp': 'str = "higher"',

'checkpoint_interval': 20000,

'continuous_eval_timeout': 3600,

'eval_tf_function': True,

'eval_tf_while_loop': False,

'loss_upper_bound': 1000000.0,

'max_to_keep': 5,

'optimizer_config': {'ema': None,

'learning_rate': {'stepwise': {'boundaries': [162400,

182700],

'name': 'PiecewiseConstantDecay',

'values': [0.0013,

0.00013,

1.3e-05]},

'type': 'stepwise'},

'optimizer': {'sgd': {'clipnorm': None,

'clipvalue': None,

'decay': 0.0,

'global_clipnorm': None,

'momentum': 0.949,

'name': 'SGD',

'nesterov': True},

'type': 'sgd'},

'warmup': {'polynomial': {'name': 'polynomial',

'power': 4,

'warmup_steps': 1000},

'type': 'polynomial'}},

'recovery_begin_steps': 0,

'recovery_max_trials': 0,

'steps_per_loop': 20000,

'summary_interval': 20000,

'train_steps': 204000,

'train_tf_function': True,

'train_tf_while_loop': True,

'validation_interval': 20000,

'validation_steps': 625}}

2021-04-07 12:20:36.378756: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set

INFO:tensorflow:Mixed precision compatibility check (mixed_float16): OK

Your GPU will likely run quickly with dtype policy mixed_float16 as it has compute capability of at least 7.0. Your GPU: GeForce RTX 2070 SUPER, compute capability 7.5

I0407 12:20:36.379258 139902763542336 device_compatibility_check.py:120] Mixed precision compatibility check (mixed_float16): OK

Your GPU will likely run quickly with dtype policy mixed_float16 as it has compute capability of at least 7.0. Your GPU: GeForce RTX 2070 SUPER, compute capability 7.5

WARNING:tensorflow:From /home/vbanna/.local/lib/python3.8/site-packages/tensorflow/python/keras/mixed_precision/loss_scale.py:56: DynamicLossScale.__init__ (from tensorflow.python.training.experimental.loss_scale) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.keras.mixed_precision.LossScaleOptimizer instead. LossScaleOptimizer now has all the functionality of DynamicLossScale

W0407 12:20:36.379376 139902763542336 deprecation.py:333] From /home/vbanna/.local/lib/python3.8/site-packages/tensorflow/python/keras/mixed_precision/loss_scale.py:56: DynamicLossScale.__init__ (from tensorflow.python.training.experimental.loss_scale) is deprecated and will be removed in a future version.

Instructions for updating:

Use tf.keras.mixed_precision.LossScaleOptimizer instead. LossScaleOptimizer now has all the functionality of DynamicLossScale

INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)

I0407 12:20:36.382394 139902763542336 mirrored_strategy.py:350] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)

I0407 12:20:36.388130 139902763542336 train_utils.py:214] Running default trainer.

2021-04-07 12:20:36.392519: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

I0407 12:20:36.443452 139902763542336 cross_device_ops.py:563] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

I0407 12:20:36.444284 139902763542336 cross_device_ops.py:563] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

I0407 12:20:36.445311 139902763542336 cross_device_ops.py:563] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

I0407 12:20:36.445759 139902763542336 cross_device_ops.py:563] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

I0407 12:20:36.513106 139902763542336 cross_device_ops.py:563] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

I0407 12:20:36.515117 139902763542336 cross_device_ops.py:563] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

I0407 12:20:36.586191 139902763542336 cross_device_ops.py:563] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

I0407 12:20:36.586790 139902763542336 cross_device_ops.py:563] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

I0407 12:20:36.587649 139902763542336 cross_device_ops.py:563] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

I0407 12:20:36.588074 139902763542336 cross_device_ops.py:563] Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

!--PREPPING GPU--!

1 Physical GPUs, 1 Logical GPUs

yolo_custom

{'task': {'init_checkpoint': 'gs://tensorflow2/darknet/cspdarknet53-golden', 'model': {'num_classes': 91, '_input_size': None, 'min_level': 3, 'max_level': 5, 'boxes_per_scale': 3, 'base': {'type': None}, 'subdivisions': 1, 'use_sam': False, 'filter': {'min_level': 3, 'max_level': 5, 'ignore_thresh': {'3': 0.7, '4': 0.7, '5': 0.7}, 'truth_thresh': {'3': 1.0, '4': 1.0, '5': 1.0}, 'loss_type': {'3': 'ciou', '4': 'ciou', '5': 'ciou'}, 'iou_normalizer': {'3': 0.05, '4': 0.05, '5': 0.05}, 'cls_normalizer': {'3': 0.5, '4': 0.5, '5': 0.5}, 'obj_normalizer': {'3': 4.0, '4': 1.0, '5': 0.4}, 'max_delta': {'3': inf, '4': inf, '5': inf}, 'new_cords': {'3': True, '4': True, '5': True}, 'scale_xy': {'3': 2.0, '4': 2.0, '5': 2.0}, 'path_scales': {'3': 8, '4': 16, '5': 32}, 'objectness_smooth': {'3': 0.0, '4': 0.0, '5': 0.0}, 'use_nms': True, 'iou_thresh': 0.001, 'nms_thresh': 0.6, 'max_boxes': 200, 'anchor_generation_scale': 512, 'use_reduction_sum': False}, 'norm_activation': {'activation': 'mish', 'use_sync_bn': True, 'norm_momentum': 0.99, 'norm_epsilon': 0.001}, 'decoder_activation': 'leaky', '_boxes': ['[12.0, 16.0]', '[19.0, 36.0]', '[40.0, 28.0]', '[36.0, 75.0]', '[76.0, 55.0]', '[72.0, 146.0]', '[142.0, 110.0]', '[192.0, 243.0]', '[459.0, 401.0]']}, 'train_data': {'input_path': 'gs://tensorflow2/coco_records/train/2017*', 'tfds_name': None, 'tfds_split': None, 'global_batch_size': 4, 'is_training': True, 'drop_remainder': True, 'shuffle_buffer_size': 10000, 'cache': False, 'cycle_length': None, 'block_length': 1, 'deterministic': None, 'sharding': True, 'enable_tf_data_service': False, 'tf_data_service_address': None, 'tf_data_service_job_name': None, 'tfds_data_dir': '', 'tfds_download': True, 'tfds_as_supervised': False, 'tfds_skip_decoding_feature': '', 'dtype': 'float16', 'decoder': {'type': 'simple_decoder', 'simple_decoder': {'regenerate_source_id': False}}, 'parser': {'image_w': 512, 'image_h': 512, 'fixed_size': True, 'max_num_instances': 200, 'min_process_size': 512, 'letter_box': False, 'random_flip': True, 'pct_rand': 0.0, 'jitter_im': 0.3, 'jitter_boxes': 0.4, 'aug_rand_translate': 0.2, 'aug_rand_saturation': 1.5, 'aug_rand_brightness': 1.5, 'aug_rand_zoom': 0.5, 'aug_rand_hue': 0.1, 'aug_rand_angle': 0.0, 'use_tie_breaker': True, 'use_scale_xy': False, 'mosaic': {'output_size': [640, 640], 'mosaic_frequency': 0.85, 'crop_area': [0.25, 1.0], 'crop_area_mosaic': [0.25, 0.75], 'random_crop': True, 'random_crop_mosaic': False}}}, 'validation_data': {'input_path': 'gs://tensorflow2/coco_records/val/2017*', 'tfds_name': None, 'tfds_split': None, 'global_batch_size': 8, 'is_training': False, 'drop_remainder': True, 'shuffle_buffer_size': 10000, 'cache': False, 'cycle_length': None, 'block_length': 1, 'deterministic': None, 'sharding': True, 'enable_tf_data_service': False, 'tf_data_service_address': None, 'tf_data_service_job_name': None, 'tfds_data_dir': '', 'tfds_download': True, 'tfds_as_supervised': False, 'tfds_skip_decoding_feature': '', 'dtype': 'float16', 'decoder': {'type': 'simple_decoder', 'simple_decoder': {'regenerate_source_id': False}}, 'parser': {'image_w': 608, 'image_h': 608, 'fixed_size': True, 'max_num_instances': 200, 'min_process_size': 320, 'letter_box': False, 'random_flip': True, 'pct_rand': 0.0, 'jitter_im': 0.3, 'jitter_boxes': 0.6, 'aug_rand_translate': 0.35, 'aug_rand_saturation': 1.5, 'aug_rand_brightness': 1.5, 'aug_rand_zoom': 0.5, 'aug_rand_hue': 0.1, 'aug_rand_angle': 0.0, 'use_tie_breaker': True, 'use_scale_xy': True, 'mosaic': {'output_size': [640, 640], 'mosaic_frequency': 0.85, 'crop_area': [0.25, 1.0], 'crop_area_mosaic': [0.25, 0.75], 'random_crop': False, 'random_crop_mosaic': False}}}, 'weight_decay': 0.001, 'annotation_file': None, 'gradient_clip_norm': 0.0, 'per_category_metrics': False, 'load_darknet_weights': False, 'darknet_load_decoder': False, 'init_checkpoint_modules': 'backbone'}, 'trainer': {'optimizer_config': {'optimizer': {'type': 'sgd', 'sgd': {'clipnorm': None, 'clipvalue': None, 'global_clipnorm': None, 'name': 'SGD', 'decay': 0.0, 'nesterov': True, 'momentum': 0.949}}, 'ema': None, 'learning_rate': {'type': 'stepwise', 'stepwise': {'name': 'PiecewiseConstantDecay', 'boundaries': [162400, 182700], 'values': [0.0013, 0.00013, 1.3e-05]}}, 'warmup': {'type': 'polynomial', 'polynomial': {'name': 'polynomial', 'power': 4, 'warmup_steps': 1000}}}, 'train_tf_while_loop': True, 'train_tf_function': True, 'eval_tf_function': True, 'eval_tf_while_loop': False, 'allow_tpu_summary': False, 'steps_per_loop': 20000, 'summary_interval': 20000, 'checkpoint_interval': 20000, 'max_to_keep': 5, 'continuous_eval_timeout': 3600, 'train_steps': 204000, 'validation_steps': 625, 'validation_interval': 20000, 'best_checkpoint_export_subdir': '', 'best_checkpoint_eval_metric': 'str = "AP50"', 'best_checkpoint_metric_comp': 'str = "higher"', 'loss_upper_bound': 1000000.0, 'recovery_begin_steps': 0, 'recovery_max_trials': 0}, 'runtime': {'distribution_strategy': 'mirrored', 'enable_xla': False, 'gpu_thread_mode': None, 'dataset_num_private_threads': None, 'per_gpu_thread_count': 0, 'tpu': None, 'num_gpus': 1, 'worker_hosts': None, 'task_index': -1, 'all_reduce_alg': None, 'num_packs': 1, 'mixed_precision_dtype': 'float16', 'loss_scale': 'dynamic', 'run_eagerly': False, 'batchnorm_spatial_persistent': False, 'num_cores_per_replica': 1, 'default_shard_dim': -1}}

{'3': 2.0, '4': 2.0, '5': 2.0} 0.0005

{'num_classes': 91, '_input_size': None, 'min_level': 3, 'max_level': 5, 'boxes_per_scale': 3, 'base': {'type': None}, 'subdivisions': 1, 'use_sam': False, 'filter': {'min_level': 3, 'max_level': 5, 'ignore_thresh': {'3': 0.7, '4': 0.7, '5': 0.7}, 'truth_thresh': {'3': 1.0, '4': 1.0, '5': 1.0}, 'loss_type': {'3': 'ciou', '4': 'ciou', '5': 'ciou'}, 'iou_normalizer': {'3': 0.05, '4': 0.05, '5': 0.05}, 'cls_normalizer': {'3': 0.5, '4': 0.5, '5': 0.5}, 'obj_normalizer': {'3': 4.0, '4': 1.0, '5': 0.4}, 'max_delta': {'3': inf, '4': inf, '5': inf}, 'new_cords': {'3': True, '4': True, '5': True}, 'scale_xy': {'3': 2.0, '4': 2.0, '5': 2.0}, 'path_scales': {'3': 8, '4': 16, '5': 32}, 'objectness_smooth': {'3': 0.0, '4': 0.0, '5': 0.0}, 'use_nms': True, 'iou_thresh': 0.001, 'nms_thresh': 0.6, 'max_boxes': 200, 'anchor_generation_scale': 512, 'use_reduction_sum': False}, 'norm_activation': {'activation': 'mish', 'use_sync_bn': True, 'norm_momentum': 0.99, 'norm_epsilon': 0.001}, 'decoder_activation': 'leaky', '_boxes': ['[12.0, 16.0]', '[19.0, 36.0]', '[40.0, 28.0]', '[36.0, 75.0]', '[76.0, 55.0]', '[72.0, 146.0]', '[142.0, 110.0]', '[192.0, 243.0]', '[459.0, 401.0]']}

InputSpec(shape=(None, None, None, 3), ndim=4)

<tensorflow.python.keras.regularizers.L2 object at 0x7f3aef575f70>

Model: "cspdarknet53"

__________________________________________________________________________________________________

Layer (type) Output Shape Param # Connected to

==================================================================================================

input_1 (InputLayer) [(None, None, None, 0

__________________________________________________________________________________________________

ConvBN_0_0 (ConvBN) (None, None, None, 3 992 input_1[0][0]

__________________________________________________________________________________________________

DarkRes_1_csp_down (CSPRoute) ((None, None, None, 27392 ConvBN_0_0[0][0]

__________________________________________________________________________________________________

DarkRes_1_0 (DarkResidual) (None, None, None, 6 20864 DarkRes_1_csp_down[0][0]

__________________________________________________________________________________________________

DarkRes_1_csp_connect (CSPConne (None, None, None, 6 12800 DarkRes_1_0[0][0]

DarkRes_1_csp_down[0][1]

__________________________________________________________________________________________________

DarkRes_2_csp_down (CSPRoute) ((None, None, None, 91136 DarkRes_1_csp_connect[0][0] WARNING:tensorflow:tf.keras.mixed_precision.experimental.LossScaleOptimizer is deprecated. Please use tf.keras.mixed_precision.LossScaleOptimizer instead. For example

opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt)

W0407 12:20:41.330750 139902763542336 loss_scale_optimizer.py:1039] tf.keras.mixed_precision.experimental.LossScaleOptimizer is deprecated. Please use tf.keras.mixed_precision.LossScaleOptimizer instead. For example

opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt)

__________________________________________________________________________________________________

DarkRes_2_0 (DarkResidual) (None, None, None, 6 41472 DarkRes_2_csp_down[0][0]

__________________________________________________________________________________________________

DarkRes_2_1 (DarkResidual) (None, None, None, 6 41472 DarkRes_2_0[0][0]

__________________________________________________________________________________________________

DarkRes_2_csp_connect (CSPConne (None, None, None, 1 21248 DarkRes_2_1[0][0]

DarkRes_2_csp_down[0][1]

__________________________________________________________________________________________________

DarkRes_3_csp_down (CSPRoute) ((None, None, None, 362496 DarkRes_2_csp_connect[0][0]

__________________________________________________________________________________________________

DarkRes_3_0 (DarkResidual) (None, None, None, 1 164864 DarkRes_3_csp_down[0][0]

__________________________________________________________________________________________________

DarkRes_3_1 (DarkResidual) (None, None, None, 1 164864 DarkRes_3_0[0][0]

__________________________________________________________________________________________________

DarkRes_3_2 (DarkResidual) (None, None, None, 1 164864 DarkRes_3_1[0][0]

__________________________________________________________________________________________________

DarkRes_3_3 (DarkResidual) (None, None, None, 1 164864 DarkRes_3_2[0][0]

__________________________________________________________________________________________________

DarkRes_3_4 (DarkResidual) (None, None, None, 1 164864 DarkRes_3_3[0][0]

__________________________________________________________________________________________________

DarkRes_3_5 (DarkResidual) (None, None, None, 1 164864 DarkRes_3_4[0][0]

__________________________________________________________________________________________________

DarkRes_3_6 (DarkResidual) (None, None, None, 1 164864 DarkRes_3_5[0][0]

__________________________________________________________________________________________________

DarkRes_3_7 (DarkResidual) (None, None, None, 1 164864 DarkRes_3_6[0][0]

__________________________________________________________________________________________________

DarkRes_3_csp_connect (CSPConne (None, None, None, 2 83456 DarkRes_3_7[0][0]

DarkRes_3_csp_down[0][1]

__________________________________________________________________________________________________

DarkRes_4_csp_down (CSPRoute) ((None, None, None, 1445888 DarkRes_3_csp_connect[0][0]

__________________________________________________________________________________________________

DarkRes_4_0 (DarkResidual) (None, None, None, 2 657408 DarkRes_4_csp_down[0][0]

__________________________________________________________________________________________________

DarkRes_4_1 (DarkResidual) (None, None, None, 2 657408 DarkRes_4_0[0][0]

__________________________________________________________________________________________________

DarkRes_4_2 (DarkResidual) (None, None, None, 2 657408 DarkRes_4_1[0][0]

__________________________________________________________________________________________________

DarkRes_4_3 (DarkResidual) (None, None, None, 2 657408 DarkRes_4_2[0][0]

__________________________________________________________________________________________________

DarkRes_4_4 (DarkResidual) (None, None, None, 2 657408 DarkRes_4_3[0][0]

__________________________________________________________________________________________________

DarkRes_4_5 (DarkResidual) (None, None, None, 2 657408 DarkRes_4_4[0][0]

__________________________________________________________________________________________________

DarkRes_4_6 (DarkResidual) (None, None, None, 2 657408 DarkRes_4_5[0][0]

__________________________________________________________________________________________________

DarkRes_4_7 (DarkResidual) (None, None, None, 2 657408 DarkRes_4_6[0][0]

__________________________________________________________________________________________________

DarkRes_4_csp_connect (CSPConne (None, None, None, 5 330752 DarkRes_4_7[0][0]

DarkRes_4_csp_down[0][1]

__________________________________________________________________________________________________

DarkRes_5_csp_down (CSPRoute) ((None, None, None, 5775360 DarkRes_4_csp_connect[0][0]

__________________________________________________________________________________________________

DarkRes_5_0 (DarkResidual) (None, None, None, 5 2625536 DarkRes_5_csp_down[0][0]

__________________________________________________________________________________________________

DarkRes_5_1 (DarkResidual) (None, None, None, 5 2625536 DarkRes_5_0[0][0]

__________________________________________________________________________________________________

DarkRes_5_2 (DarkResidual) (None, None, None, 5 2625536 DarkRes_5_1[0][0]

__________________________________________________________________________________________________

DarkRes_5_3 (DarkResidual) (None, None, None, 5 2625536 DarkRes_5_2[0][0]

__________________________________________________________________________________________________

DarkRes_5_csp_connect (CSPConne (None, None, None, 1 1316864 DarkRes_5_3[0][0]

DarkRes_5_csp_down[0][1]

==================================================================================================

Total params: 26,652,512

Trainable params: 26,617,184

Non-trainable params: 35,328

__________________________________________________________________________________________________

conv1 512.0

conv2 1024.0

conv1 512.0

spp

conv1 512.0

conv2 1024.0

conv1 512.0

7

conv1 512.0

conv2 256.0

conv1 512.0

conv2 256.0

4

ListWrapper([256, 512, 1024])

conv1 128

conv2 256

conv1 128

conv2 256

conv1 128

conv2 256

6

conv1 256

conv2 512

conv1 256

conv2 512

conv1 256

conv2 512

6

conv1 512

conv2 1024

conv1 512

conv2 1024

conv1 512

conv2 1024

6

detget {'3': 2.0, '4': 2.0, '5': 2.0}

Model: "yolo"

_________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

cspdarknet53 (Darknet) OrderedDict([('3', (None, 26652512

_________________________________________________________________

YoloDecoder (YoloDecoder) {'3': (None, None, None, 37319168

_________________________________________________________________

yolo_head (YoloHead) multiple 516960

_________________________________________________________________

yolo_layer (YoloLayer) multiple 0

=================================================================

Total params: 64,488,640

Trainable params: 64,422,336

Non-trainable params: 66,304

_________________________________________________________________

{'3': 2.0, '4': 2.0, '5': 2.0}

<PrefetchDataset shapes: ((8, 608, 608, 3), {source_id: (8,), bbox: (8, 200, 4), classes: (8, 200), area: (8, 200), is_crowd: (8, 200), best_anchors: (8, 200, 9), best_iou_match: (8, 200, 9), width: (8,), height: (8,), info: (8, 4), num_detections: (8,), upds: {3: (8, 200, 8), 4: (8, 200, 8), 5: (8, 200, 8)}, inds: {3: (8, 200, 3), 4: (8, 200, 3), 5: (8, 200, 3)}, true_conf: {3: (8, 76, 76, 3, 1), 4: (8, 38, 38, 3, 1), 5: (8, 19, 19, 3, 1)}}), types: (tf.float16, {source_id: tf.int64, bbox: tf.float16, classes: tf.float16, area: tf.float16, is_crowd: tf.int32, best_anchors: tf.float16, best_iou_match: tf.float32, width: tf.int32, height: tf.int32, info: tf.int32, num_detections: tf.int32, upds: {3: tf.float16, 4: tf.float16, 5: tf.float16}, inds: {3: tf.int32, 4: tf.int32, 5: tf.int32}, true_conf: {3: tf.float16, 4: tf.float16, 5: tf.float16}})>I0407 12:20:47.385322 139902763542336 controller.py:362] restoring or initializing model...

Traceback (most recent call last):

File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main

return _run_code(code, main_globals, None,

File "/usr/lib/python3.8/runpy.py", line 85, in _run_code

exec(code, run_globals)

File "/home/vbanna/Desktop/Research/TensorFlowModelGardeners/yolo/train.py", line 156, in <module>

app.run(main)

File "/home/vbanna/.local/lib/python3.8/site-packages/absl/app.py", line 300, in run

_run_main(main, args)

File "/home/vbanna/.local/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main

sys.exit(main(argv))

File "/home/vbanna/Desktop/Research/TensorFlowModelGardeners/yolo/train.py", line 143, in main

train_lib.run_experiment(

File "/home/vbanna/Desktop/Research/TensorFlowModelGardeners/official/core/train_lib.py", line 83, in run_experiment

controller = orbit.Controller(

File "/home/vbanna/Desktop/Research/TensorFlowModelGardeners/orbit/controller.py", line 190, in __init__

restored_path = self.restore_checkpoint()

File "/home/vbanna/Desktop/Research/TensorFlowModelGardeners/orbit/controller.py", line 363, in restore_checkpoint

checkpoint_path = self.checkpoint_manager.restore_or_initialize()

File "/home/vbanna/.local/lib/python3.8/site-packages/tensorflow/python/training/checkpoint_management.py", line 851, in restore_or_initialize

self._init_fn()

File "/home/vbanna/Desktop/Research/TensorFlowModelGardeners/official/core/base_trainer.py", line 255, in initialize

self.task.initialize(self.model)

File "/home/vbanna/Desktop/Research/TensorFlowModelGardeners/yolo/tasks/yolo.py", line 429, in initialize

if tf.io.gfile.isdir(ckpt_dir_or_file):

File "/home/vbanna/.local/lib/python3.8/site-packages/tensorflow/python/lib/io/file_io.py", line 629, in is_directory_v2

return _pywrap_file_io.IsDirectory(compat.path_to_bytes(path))

KeyboardInterrupt

restoring or initializing model...