2015-04-19 08:22:51,141 INFO repeater task ashtech_hcorrstats starting 3.0 seconds late 2015-04-19 08:22:52,217 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 08:22:57,223 INFO repeater task os75nb_lastensq starting 0.2 seconds late 2015-04-19 08:22:59,783 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 08:22:59,784 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 08:22:47 2015-04-19 08:23:29,827 INFO repeater task os75nb_10pings starting 1.7 seconds late 2015-04-19 08:23:30,022 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 08:27:50,383 INFO repeater task os75nb_quick starting 2.2 seconds late 2015-04-19 08:27:57,420 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 08:27:57,421 INFO repeater task os75nb_3dayplots starting 9.2 seconds late 2015-04-19 08:28:10,665 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 08:28:10,666 INFO repeater task os75nb_3dayplots ended after 13.2 seconds 2015-04-19 08:28:10,666 INFO repeater task os75nb_tsstats starting 22.5 seconds late 2015-04-19 08:28:11,776 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 08:28:11,776 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 08:28:11,777 INFO repeater task os75nb_beamplots starting 23.6 seconds late 2015-04-19 08:28:13,493 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 08:28:13,493 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 08:28:13,494 INFO repeater task os75nb_beamplots starting 25.3 seconds late 2015-04-19 08:28:15,225 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 08:28:15,226 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 08:28:15,226 INFO repeater task ashtech_hcorrstats starting 27.0 seconds late 2015-04-19 08:28:16,313 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 08:28:21,319 INFO repeater task os75nb_lastensq starting 24.3 seconds late 2015-04-19 08:28:23,876 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 08:28:23,876 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 08:27:47 2015-04-19 08:28:48,914 INFO repeater task os75nb_10pings starting 0.7 seconds late 2015-04-19 08:28:49,099 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 08:32:49,467 INFO repeater task ashtech_hcorrstats starting 1.3 seconds late 2015-04-19 08:32:50,550 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 08:33:00,563 INFO repeater task os75nb_lastensq starting 3.6 seconds late 2015-04-19 08:33:03,128 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 08:33:03,129 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 08:32:47 2015-04-19 08:34:08,227 INFO repeater task os75nb_10pings starting 0.1 seconds late 2015-04-19 08:34:08,409 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 08:37:48,731 INFO repeater task ashtech_hcorrstats starting 0.6 seconds late 2015-04-19 08:37:49,823 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 08:37:59,836 INFO repeater task os75nb_lastensq starting 2.8 seconds late 2015-04-19 08:38:02,402 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 08:38:02,403 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 08:37:47 2015-04-19 08:39:32,524 INFO repeater task os75nb_10pings starting 4.4 seconds late 2015-04-19 08:39:32,709 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 08:42:52,952 INFO repeater task os75nb_quick starting 4.8 seconds late 2015-04-19 08:42:59,962 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 08:42:59,963 INFO repeater task os75nb_3dayplots starting 11.8 seconds late 2015-04-19 08:43:13,018 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 08:43:13,018 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2015-04-19 08:43:13,019 INFO repeater task os75nb_tsstats starting 24.8 seconds late 2015-04-19 08:43:14,125 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 08:43:14,126 INFO repeater task os75nb_beamplots starting 26.0 seconds late 2015-04-19 08:43:15,847 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 08:43:15,848 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 08:43:15,848 INFO repeater task os75nb_beamplots starting 27.7 seconds late 2015-04-19 08:43:17,574 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 08:43:17,574 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 08:43:17,574 INFO repeater task ashtech_hcorrstats starting 29.4 seconds late 2015-04-19 08:43:18,647 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 08:43:23,650 INFO repeater task os75nb_lastensq starting 26.7 seconds late 2015-04-19 08:43:26,218 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 08:43:26,218 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 08:42:47 2015-04-19 08:44:51,341 INFO repeater task os75nb_10pings starting 3.2 seconds late 2015-04-19 08:44:51,524 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 08:47:51,788 INFO repeater task ashtech_hcorrstats starting 3.6 seconds late 2015-04-19 08:47:52,872 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 08:47:57,878 INFO repeater task os75nb_lastensq starting 0.9 seconds late 2015-04-19 08:48:00,441 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 08:48:00,442 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 08:47:47 2015-04-19 08:50:10,629 INFO repeater task os75nb_10pings starting 2.5 seconds late 2015-04-19 08:50:10,811 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 08:52:51,034 INFO repeater task ashtech_hcorrstats starting 2.9 seconds late 2015-04-19 08:52:52,126 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 08:52:57,132 INFO repeater task os75nb_lastensq starting 0.1 seconds late 2015-04-19 08:52:59,713 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 08:52:59,714 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 08:52:47 2015-04-19 08:55:29,913 INFO repeater task os75nb_10pings starting 1.7 seconds late 2015-04-19 08:55:30,094 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 08:57:50,272 INFO repeater task os75nb_quick starting 2.1 seconds late 2015-04-19 08:57:57,327 INFO repeater task os75nb_quick ended after 7.1 seconds 2015-04-19 08:57:57,327 INFO repeater task os75nb_3dayplots starting 9.2 seconds late 2015-04-19 08:58:10,532 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 08:58:10,532 INFO repeater task os75nb_3dayplots ended after 13.2 seconds 2015-04-19 08:58:10,532 INFO repeater task os75nb_tsstats starting 22.4 seconds late 2015-04-19 08:58:11,621 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 08:58:11,621 INFO repeater task os75nb_beamplots starting 23.4 seconds late 2015-04-19 08:58:13,334 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 08:58:13,334 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 08:58:13,335 INFO repeater task os75nb_beamplots starting 25.2 seconds late 2015-04-19 08:58:15,070 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 08:58:15,070 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 08:58:15,071 INFO repeater task ashtech_hcorrstats starting 26.9 seconds late 2015-04-19 08:58:16,145 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 08:58:21,151 INFO repeater task os75nb_lastensq starting 24.2 seconds late 2015-04-19 08:58:23,703 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 08:58:23,703 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 08:57:47 2015-04-19 09:00:48,877 INFO repeater task os75nb_10pings starting 0.7 seconds late 2015-04-19 09:00:49,065 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 09:02:49,263 INFO repeater task ashtech_hcorrstats starting 1.1 seconds late 2015-04-19 09:02:50,334 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:03:00,347 INFO repeater task os75nb_lastensq starting 3.3 seconds late 2015-04-19 09:03:02,907 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:03:02,907 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:02:47 2015-04-19 09:06:13,145 INFO repeater task os75nb_10pings starting 5.0 seconds late 2015-04-19 09:06:13,331 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 09:07:48,455 INFO repeater task ashtech_hcorrstats starting 0.3 seconds late 2015-04-19 09:07:49,540 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:07:59,552 INFO repeater task os75nb_lastensq starting 2.6 seconds late 2015-04-19 09:08:02,144 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:08:02,144 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:07:47 2015-04-19 09:11:32,408 INFO repeater task os75nb_10pings starting 4.2 seconds late 2015-04-19 09:11:32,596 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 09:12:52,699 INFO repeater task os75nb_quick starting 4.5 seconds late 2015-04-19 09:12:59,689 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 09:12:59,689 INFO repeater task os75nb_3dayplots starting 11.5 seconds late 2015-04-19 09:13:12,698 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 09:13:12,698 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2015-04-19 09:13:12,699 INFO repeater task os75nb_tsstats starting 24.5 seconds late 2015-04-19 09:13:13,795 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 09:13:13,796 INFO repeater task os75nb_beamplots starting 25.6 seconds late 2015-04-19 09:13:15,531 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 09:13:15,532 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 09:13:15,532 INFO repeater task os75nb_beamplots starting 27.4 seconds late 2015-04-19 09:13:17,269 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 09:13:17,269 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 09:13:17,269 INFO repeater task ashtech_hcorrstats starting 29.1 seconds late 2015-04-19 09:13:18,336 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:13:23,342 INFO repeater task os75nb_lastensq starting 26.3 seconds late 2015-04-19 09:13:25,912 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:13:25,912 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:12:47 2015-04-19 09:16:51,171 INFO repeater task os75nb_10pings starting 3.0 seconds late 2015-04-19 09:16:51,357 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 09:17:51,441 INFO repeater task ashtech_hcorrstats starting 3.3 seconds late 2015-04-19 09:17:52,518 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:17:57,524 INFO repeater task os75nb_lastensq starting 0.5 seconds late 2015-04-19 09:18:00,084 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:18:00,085 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:17:47 2015-04-19 09:22:10,433 INFO repeater task os75nb_10pings starting 2.3 seconds late 2015-04-19 09:22:10,617 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 09:22:50,675 INFO repeater task ashtech_hcorrstats starting 2.5 seconds late 2015-04-19 09:22:51,756 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:23:01,769 INFO repeater task os75nb_lastensq starting 4.8 seconds late 2015-04-19 09:23:04,336 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:23:04,336 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:22:47 2015-04-19 09:27:29,716 INFO repeater task os75nb_10pings starting 1.5 seconds late 2015-04-19 09:27:29,901 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 09:27:49,931 INFO repeater task os75nb_quick starting 1.8 seconds late 2015-04-19 09:27:56,945 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 09:27:56,946 INFO repeater task os75nb_3dayplots starting 8.8 seconds late 2015-04-19 09:28:10,012 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) zgrid failed zgrid failed: returning all values masked 2015-04-19 09:28:10,012 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2015-04-19 09:28:10,012 INFO repeater task os75nb_tsstats starting 21.8 seconds late 2015-04-19 09:28:11,103 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 09:28:11,104 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 09:28:11,104 INFO repeater task os75nb_beamplots starting 22.9 seconds late 2015-04-19 09:28:12,852 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 09:28:12,852 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 09:28:12,852 INFO repeater task os75nb_beamplots starting 24.7 seconds late 2015-04-19 09:28:14,603 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 09:28:14,603 INFO repeater task os75nb_beamplots ended after 1.8 seconds 2015-04-19 09:28:14,603 INFO repeater task ashtech_hcorrstats starting 26.4 seconds late 2015-04-19 09:28:15,680 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:28:20,686 INFO repeater task os75nb_lastensq starting 23.7 seconds late 2015-04-19 09:28:23,270 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:28:23,271 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:27:47 2015-04-19 09:32:48,648 INFO repeater task os75nb_10pings starting 0.5 seconds late 2015-04-19 09:32:48,833 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 09:32:48,834 INFO repeater task ashtech_hcorrstats starting 0.7 seconds late 2015-04-19 09:32:49,914 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:32:59,927 INFO repeater task os75nb_lastensq starting 2.9 seconds late 2015-04-19 09:33:02,480 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:33:02,480 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:32:47 2015-04-19 09:37:52,906 INFO repeater task ashtech_hcorrstats starting 4.7 seconds late 2015-04-19 09:37:53,976 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:37:58,982 INFO repeater task os75nb_lastensq starting 2.0 seconds late 2015-04-19 09:38:01,558 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:38:01,558 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:37:47 2015-04-19 09:38:11,573 INFO repeater task os75nb_10pings starting 3.4 seconds late 2015-04-19 09:38:11,757 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 09:42:52,143 INFO repeater task os75nb_quick starting 4.0 seconds late 2015-04-19 09:42:59,182 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 09:42:59,182 INFO repeater task os75nb_3dayplots starting 11.0 seconds late 2015-04-19 09:43:12,296 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) zgrid failed zgrid failed: returning all values masked zgrid failed zgrid failed: returning all values masked 2015-04-19 09:43:12,296 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2015-04-19 09:43:12,297 INFO repeater task os75nb_tsstats starting 24.1 seconds late 2015-04-19 09:43:13,393 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 09:43:13,394 INFO repeater task os75nb_beamplots starting 25.2 seconds late 2015-04-19 09:43:15,110 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 09:43:15,110 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 09:43:15,110 INFO repeater task os75nb_beamplots starting 26.9 seconds late 2015-04-19 09:43:16,846 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 09:43:16,846 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 09:43:16,846 INFO repeater task ashtech_hcorrstats starting 28.7 seconds late 2015-04-19 09:43:17,920 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:43:22,926 INFO repeater task os75nb_lastensq starting 25.9 seconds late 2015-04-19 09:43:25,482 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:43:25,482 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:42:47 2015-04-19 09:43:30,484 INFO repeater task os75nb_10pings starting 2.3 seconds late 2015-04-19 09:43:30,669 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 09:47:51,034 INFO repeater task ashtech_hcorrstats starting 2.9 seconds late 2015-04-19 09:47:52,109 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:47:57,114 INFO repeater task os75nb_lastensq starting 0.1 seconds late 2015-04-19 09:47:59,686 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:47:59,686 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:47:47 2015-04-19 09:48:49,752 INFO repeater task os75nb_10pings starting 1.6 seconds late 2015-04-19 09:48:49,937 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 09:52:50,227 INFO repeater task ashtech_hcorrstats starting 2.0 seconds late 2015-04-19 09:52:51,296 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:53:01,307 INFO repeater task os75nb_lastensq starting 4.3 seconds late 2015-04-19 09:53:03,897 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:53:03,897 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:52:47 2015-04-19 09:54:08,987 INFO repeater task os75nb_10pings starting 0.8 seconds late 2015-04-19 09:54:09,174 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 09:57:49,465 INFO repeater task os75nb_quick starting 1.3 seconds late 2015-04-19 09:57:56,476 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 09:57:56,476 INFO repeater task os75nb_3dayplots starting 8.3 seconds late 2015-04-19 09:58:09,544 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) zgrid failed zgrid failed: returning all values masked 2015-04-19 09:58:09,544 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2015-04-19 09:58:09,545 INFO repeater task os75nb_tsstats starting 21.4 seconds late 2015-04-19 09:58:10,654 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 09:58:10,655 INFO repeater task os75nb_beamplots starting 22.5 seconds late 2015-04-19 09:58:12,389 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 09:58:12,389 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 09:58:12,389 INFO repeater task os75nb_beamplots starting 24.2 seconds late 2015-04-19 09:58:14,119 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 09:58:14,120 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 09:58:14,120 INFO repeater task ashtech_hcorrstats starting 25.9 seconds late 2015-04-19 09:58:15,193 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 09:58:20,199 INFO repeater task os75nb_lastensq starting 23.2 seconds late 2015-04-19 09:58:22,783 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 09:58:22,783 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 09:57:47 2015-04-19 09:59:32,884 INFO repeater task os75nb_10pings starting 4.7 seconds late 2015-04-19 09:59:33,069 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 10:02:48,342 INFO repeater task ashtech_hcorrstats starting 0.2 seconds late 2015-04-19 10:02:49,418 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:02:59,431 INFO repeater task os75nb_lastensq starting 2.4 seconds late 2015-04-19 10:03:02,024 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:03:02,024 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:02:47 2015-04-19 10:04:52,147 INFO repeater task os75nb_10pings starting 4.0 seconds late 2015-04-19 10:04:52,330 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 10:07:52,555 INFO repeater task ashtech_hcorrstats starting 4.4 seconds late 2015-04-19 10:07:53,628 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:07:58,634 INFO repeater task os75nb_lastensq starting 1.6 seconds late 2015-04-19 10:08:01,214 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:08:01,214 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:07:47 2015-04-19 10:10:11,398 INFO repeater task os75nb_10pings starting 3.2 seconds late 2015-04-19 10:10:11,583 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 10:12:51,792 INFO repeater task os75nb_quick starting 3.6 seconds late 2015-04-19 10:12:58,829 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 10:12:58,829 INFO repeater task os75nb_3dayplots starting 10.7 seconds late 2015-04-19 10:13:11,827 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) zgrid failed zgrid failed: returning all values masked 2015-04-19 10:13:11,827 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2015-04-19 10:13:11,827 INFO repeater task os75nb_tsstats starting 23.7 seconds late 2015-04-19 10:13:12,935 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 10:13:12,936 INFO repeater task os75nb_beamplots starting 24.8 seconds late 2015-04-19 10:13:14,673 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 10:13:14,673 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 10:13:14,674 INFO repeater task os75nb_beamplots starting 26.5 seconds late 2015-04-19 10:13:16,419 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 10:13:16,420 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 10:13:16,420 INFO repeater task ashtech_hcorrstats starting 28.2 seconds late 2015-04-19 10:13:17,483 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:13:22,488 INFO repeater task os75nb_lastensq starting 25.5 seconds late 2015-04-19 10:13:25,046 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:13:25,046 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:12:47 2015-04-19 10:15:30,211 INFO repeater task os75nb_10pings starting 2.0 seconds late 2015-04-19 10:15:30,394 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 10:17:50,595 INFO repeater task ashtech_hcorrstats starting 2.4 seconds late 2015-04-19 10:17:51,669 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:18:01,683 INFO repeater task os75nb_lastensq starting 4.7 seconds late 2015-04-19 10:18:04,234 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:18:04,234 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:17:47 2015-04-19 10:20:49,437 INFO repeater task os75nb_10pings starting 1.3 seconds late 2015-04-19 10:20:49,623 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 10:22:49,775 INFO repeater task ashtech_hcorrstats starting 1.6 seconds late 2015-04-19 10:22:50,854 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:23:00,868 INFO repeater task os75nb_lastensq starting 3.9 seconds late 2015-04-19 10:23:03,451 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:23:03,451 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:22:47 2015-04-19 10:26:08,699 INFO repeater task os75nb_10pings starting 0.5 seconds late 2015-04-19 10:26:08,896 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 10:27:49,039 INFO repeater task os75nb_quick starting 0.9 seconds late 2015-04-19 10:27:56,106 INFO repeater task os75nb_quick ended after 7.1 seconds 2015-04-19 10:27:56,107 INFO repeater task os75nb_3dayplots starting 7.9 seconds late 2015-04-19 10:28:09,187 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) zgrid failed zgrid failed: returning all values masked 2015-04-19 10:28:09,187 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2015-04-19 10:28:09,188 INFO repeater task os75nb_tsstats starting 21.0 seconds late 2015-04-19 10:28:10,298 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 10:28:10,298 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 10:28:10,298 INFO repeater task os75nb_beamplots starting 22.1 seconds late 2015-04-19 10:28:12,032 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 10:28:12,032 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 10:28:12,032 INFO repeater task os75nb_beamplots starting 23.9 seconds late 2015-04-19 10:28:13,772 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 10:28:13,773 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 10:28:13,773 INFO repeater task ashtech_hcorrstats starting 25.6 seconds late 2015-04-19 10:28:14,859 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:28:19,864 INFO repeater task os75nb_lastensq starting 22.9 seconds late 2015-04-19 10:28:22,447 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:28:22,448 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:27:47 2015-04-19 10:31:32,669 INFO repeater task os75nb_10pings starting 4.5 seconds late 2015-04-19 10:31:32,852 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 10:32:52,962 INFO repeater task ashtech_hcorrstats starting 4.8 seconds late 2015-04-19 10:32:54,045 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:32:59,051 INFO repeater task os75nb_lastensq starting 2.1 seconds late 2015-04-19 10:33:01,633 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:33:01,634 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:32:47 2015-04-19 10:36:51,932 INFO repeater task os75nb_10pings starting 3.8 seconds late 2015-04-19 10:36:52,113 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 10:37:52,201 INFO repeater task ashtech_hcorrstats starting 4.0 seconds late 2015-04-19 10:37:53,274 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:37:58,278 INFO repeater task os75nb_lastensq starting 1.3 seconds late 2015-04-19 10:38:00,840 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:38:00,840 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:37:47 2015-04-19 10:42:11,193 INFO repeater task os75nb_10pings starting 3.0 seconds late 2015-04-19 10:42:11,376 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 10:42:51,432 INFO repeater task os75nb_quick starting 3.3 seconds late 2015-04-19 10:42:58,540 INFO repeater task os75nb_quick ended after 7.1 seconds 2015-04-19 10:42:58,540 INFO repeater task os75nb_3dayplots starting 10.4 seconds late 2015-04-19 10:43:11,538 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) zgrid failed zgrid failed: returning all values masked 2015-04-19 10:43:11,538 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2015-04-19 10:43:11,539 INFO repeater task os75nb_tsstats starting 23.4 seconds late 2015-04-19 10:43:12,646 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 10:43:12,647 INFO repeater task os75nb_beamplots starting 24.5 seconds late 2015-04-19 10:43:14,376 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 10:43:14,376 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 10:43:14,377 INFO repeater task os75nb_beamplots starting 26.2 seconds late 2015-04-19 10:43:16,099 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 10:43:16,099 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 10:43:16,100 INFO repeater task ashtech_hcorrstats starting 27.9 seconds late 2015-04-19 10:43:17,170 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:43:22,175 INFO repeater task os75nb_lastensq starting 25.2 seconds late 2015-04-19 10:43:24,753 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:43:24,754 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:42:47 2015-04-19 10:47:30,058 INFO repeater task os75nb_10pings starting 1.9 seconds late 2015-04-19 10:47:30,238 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 10:47:50,270 INFO repeater task ashtech_hcorrstats starting 2.1 seconds late 2015-04-19 10:47:51,340 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:48:01,353 INFO repeater task os75nb_lastensq starting 4.4 seconds late 2015-04-19 10:48:03,929 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:48:03,930 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:47:47 2015-04-19 10:52:49,321 INFO repeater task os75nb_10pings starting 1.1 seconds late 2015-04-19 10:52:49,505 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 10:52:49,506 INFO repeater task ashtech_hcorrstats starting 1.3 seconds late 2015-04-19 10:52:50,566 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:53:00,580 INFO repeater task os75nb_lastensq starting 3.6 seconds late 2015-04-19 10:53:03,133 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:53:03,133 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:52:47 2015-04-19 10:57:48,462 INFO repeater task os75nb_quick starting 0.3 seconds late 2015-04-19 10:57:55,482 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 10:57:55,482 INFO repeater task os75nb_3dayplots starting 7.3 seconds late 2015-04-19 10:58:08,531 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 10:58:08,531 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2015-04-19 10:58:08,532 INFO repeater task os75nb_tsstats starting 20.4 seconds late 2015-04-19 10:58:09,641 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 10:58:09,641 INFO repeater task os75nb_beamplots starting 21.5 seconds late 2015-04-19 10:58:11,376 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 10:58:11,376 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 10:58:11,376 INFO repeater task os75nb_beamplots starting 23.2 seconds late 2015-04-19 10:58:13,112 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 10:58:13,112 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 10:58:13,113 INFO repeater task ashtech_hcorrstats starting 24.9 seconds late 2015-04-19 10:58:14,176 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 10:58:19,182 INFO repeater task os75nb_lastensq starting 22.2 seconds late 2015-04-19 10:58:21,774 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 10:58:21,774 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 10:57:47 2015-04-19 10:58:21,775 INFO repeater task os75nb_10pings starting 13.6 seconds late 2015-04-19 10:58:21,958 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:02:52,312 INFO repeater task ashtech_hcorrstats starting 4.1 seconds late 2015-04-19 11:02:53,383 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:02:58,389 INFO repeater task os75nb_lastensq starting 1.4 seconds late 2015-04-19 11:03:00,960 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 11:03:00,960 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:02:47 2015-04-19 11:03:31,002 INFO repeater task os75nb_10pings starting 2.8 seconds late 2015-04-19 11:03:31,185 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:07:51,550 INFO repeater task ashtech_hcorrstats starting 3.4 seconds late 2015-04-19 11:07:52,627 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:07:57,632 INFO repeater task os75nb_lastensq starting 0.6 seconds late 2015-04-19 11:08:00,210 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 11:08:00,210 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:07:47 2015-04-19 11:08:50,283 INFO repeater task os75nb_10pings starting 2.1 seconds late 2015-04-19 11:08:50,470 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:12:50,786 INFO repeater task os75nb_quick starting 2.6 seconds late 2015-04-19 11:12:57,818 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 11:12:57,818 INFO repeater task os75nb_3dayplots starting 9.6 seconds late 2015-04-19 11:13:10,918 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) zgrid failed zgrid failed: returning all values masked zgrid failed zgrid failed: returning all values masked 2015-04-19 11:13:10,918 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2015-04-19 11:13:10,918 INFO repeater task os75nb_tsstats starting 22.7 seconds late 2015-04-19 11:13:12,028 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 11:13:12,029 INFO repeater task os75nb_beamplots starting 23.9 seconds late 2015-04-19 11:13:13,739 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 11:13:13,739 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 11:13:13,739 INFO repeater task os75nb_beamplots starting 25.6 seconds late 2015-04-19 11:13:15,454 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 11:13:15,455 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 11:13:15,455 INFO repeater task ashtech_hcorrstats starting 27.3 seconds late 2015-04-19 11:13:16,511 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:13:21,517 INFO repeater task os75nb_lastensq starting 24.5 seconds late 2015-04-19 11:13:24,067 INFO repeater task os75nb_lastensq ended after 2.5 seconds 2015-04-19 11:13:24,067 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:12:47 2015-04-19 11:14:09,143 INFO repeater task os75nb_10pings starting 1.0 seconds late 2015-04-19 11:14:09,330 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:17:49,629 INFO repeater task ashtech_hcorrstats starting 1.4 seconds late 2015-04-19 11:17:50,686 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:18:00,699 INFO repeater task os75nb_lastensq starting 3.7 seconds late 2015-04-19 11:18:03,300 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 11:18:03,300 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:17:47 2015-04-19 11:19:28,418 INFO repeater task os75nb_10pings starting 0.2 seconds late 2015-04-19 11:19:28,606 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:22:48,869 INFO repeater task ashtech_hcorrstats starting 0.7 seconds late 2015-04-19 11:22:49,939 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:22:59,947 INFO repeater task os75nb_lastensq starting 2.9 seconds late 2015-04-19 11:23:02,531 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 11:23:02,532 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:22:47 2015-04-19 11:24:52,694 INFO repeater task os75nb_10pings starting 4.5 seconds late 2015-04-19 11:24:52,883 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:27:53,150 INFO repeater task os75nb_quick starting 5.0 seconds late 2015-04-19 11:28:00,384 INFO repeater task os75nb_quick ended after 7.2 seconds 2015-04-19 11:28:00,384 INFO repeater task os75nb_3dayplots starting 12.2 seconds late 2015-04-19 11:28:13,335 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) zgrid failed zgrid failed: returning all values masked zgrid failed zgrid failed: returning all values masked 2015-04-19 11:28:13,335 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2015-04-19 11:28:13,336 INFO repeater task os75nb_tsstats starting 25.2 seconds late 2015-04-19 11:28:14,442 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 11:28:14,443 INFO repeater task os75nb_beamplots starting 26.3 seconds late 2015-04-19 11:28:16,174 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 11:28:16,175 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 11:28:16,175 INFO repeater task os75nb_beamplots starting 28.0 seconds late 2015-04-19 11:28:17,892 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 11:28:17,892 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 11:28:17,893 INFO repeater task ashtech_hcorrstats starting 29.7 seconds late 2015-04-19 11:28:18,956 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:28:23,959 INFO repeater task os75nb_lastensq starting 27.0 seconds late 2015-04-19 11:28:26,525 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 11:28:26,525 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:27:47 2015-04-19 11:30:11,672 INFO repeater task os75nb_10pings starting 3.5 seconds late 2015-04-19 11:30:11,855 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:32:52,056 INFO repeater task ashtech_hcorrstats starting 3.9 seconds late 2015-04-19 11:32:53,137 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:32:58,143 INFO repeater task os75nb_lastensq starting 1.1 seconds late 2015-04-19 11:33:00,704 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 11:33:00,705 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:32:47 2015-04-19 11:35:30,900 INFO repeater task os75nb_10pings starting 2.7 seconds late 2015-04-19 11:35:31,087 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:37:51,301 INFO repeater task ashtech_hcorrstats starting 3.1 seconds late 2015-04-19 11:37:52,363 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:37:57,369 INFO repeater task os75nb_lastensq starting 0.4 seconds late 2015-04-19 11:37:59,941 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 11:37:59,942 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:37:47 2015-04-19 11:40:50,171 INFO repeater task os75nb_10pings starting 2.0 seconds late 2015-04-19 11:40:50,360 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:42:50,536 INFO repeater task os75nb_quick starting 2.4 seconds late 2015-04-19 11:42:57,607 INFO repeater task os75nb_quick ended after 7.1 seconds 2015-04-19 11:42:57,607 INFO repeater task os75nb_3dayplots starting 9.4 seconds late 2015-04-19 11:43:10,679 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 11:43:10,680 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2015-04-19 11:43:10,680 INFO repeater task os75nb_tsstats starting 22.5 seconds late 2015-04-19 11:43:11,786 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 11:43:11,786 INFO repeater task os75nb_beamplots starting 23.6 seconds late 2015-04-19 11:43:13,514 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 11:43:13,515 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 11:43:13,515 INFO repeater task os75nb_beamplots starting 25.3 seconds late 2015-04-19 11:43:15,261 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 11:43:15,262 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 11:43:15,262 INFO repeater task ashtech_hcorrstats starting 27.1 seconds late 2015-04-19 11:43:16,322 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:43:21,326 INFO repeater task os75nb_lastensq starting 24.3 seconds late 2015-04-19 11:43:23,929 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 11:43:23,929 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:42:47 2015-04-19 11:46:09,165 INFO repeater task os75nb_10pings starting 1.0 seconds late 2015-04-19 11:46:09,352 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:47:49,482 INFO repeater task ashtech_hcorrstats starting 1.3 seconds late 2015-04-19 11:47:50,550 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:48:00,563 INFO repeater task os75nb_lastensq starting 3.6 seconds late 2015-04-19 11:48:03,180 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 11:48:03,181 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:47:47 2015-04-19 11:51:28,428 INFO repeater task os75nb_10pings starting 0.3 seconds late 2015-04-19 11:51:28,615 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:52:48,721 INFO repeater task ashtech_hcorrstats starting 0.5 seconds late 2015-04-19 11:52:49,778 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:52:59,787 INFO repeater task os75nb_lastensq starting 2.8 seconds late 2015-04-19 11:53:02,381 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 11:53:02,381 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:52:47 2015-04-19 11:56:52,696 INFO repeater task os75nb_10pings starting 4.5 seconds late 2015-04-19 11:56:52,881 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 11:57:52,965 INFO repeater task os75nb_quick starting 4.8 seconds late 2015-04-19 11:58:00,047 INFO repeater task os75nb_quick ended after 7.1 seconds 2015-04-19 11:58:00,048 INFO repeater task os75nb_3dayplots starting 11.9 seconds late 2015-04-19 11:58:13,061 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 11:58:13,062 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2015-04-19 11:58:13,062 INFO repeater task os75nb_tsstats starting 24.9 seconds late 2015-04-19 11:58:14,180 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 11:58:14,180 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 11:58:14,180 INFO repeater task os75nb_beamplots starting 26.0 seconds late 2015-04-19 11:58:15,911 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 11:58:15,912 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 11:58:15,912 INFO repeater task os75nb_beamplots starting 27.7 seconds late 2015-04-19 11:58:17,645 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 11:58:17,646 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 11:58:17,646 INFO repeater task ashtech_hcorrstats starting 29.5 seconds late 2015-04-19 11:58:18,701 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 11:58:23,707 INFO repeater task os75nb_lastensq starting 26.7 seconds late 2015-04-19 11:58:26,324 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 11:58:26,325 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 11:57:47 2015-04-19 12:02:11,655 INFO repeater task os75nb_10pings starting 3.5 seconds late 2015-04-19 12:02:11,843 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 12:02:51,902 INFO repeater task ashtech_hcorrstats starting 3.7 seconds late 2015-04-19 12:02:52,993 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:02:57,999 INFO repeater task os75nb_lastensq starting 1.0 seconds late 2015-04-19 12:03:00,623 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:03:00,623 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:02:47 2015-04-19 12:07:31,002 INFO repeater task os75nb_10pings starting 2.8 seconds late 2015-04-19 12:07:31,186 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 12:07:51,218 INFO repeater task ashtech_hcorrstats starting 3.0 seconds late 2015-04-19 12:07:52,332 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:07:57,338 INFO repeater task os75nb_lastensq starting 0.3 seconds late 2015-04-19 12:07:59,904 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:07:59,905 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:07:47 2015-04-19 12:12:50,286 INFO repeater task os75nb_10pings starting 2.1 seconds late 2015-04-19 12:12:50,472 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 12:12:50,472 INFO repeater task os75nb_quick starting 2.3 seconds late 2015-04-19 12:12:57,503 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 12:12:57,503 INFO repeater task os75nb_3dayplots starting 9.3 seconds late 2015-04-19 12:13:10,575 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 12:13:10,575 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2015-04-19 12:13:10,575 INFO repeater task os75nb_tsstats starting 22.4 seconds late 2015-04-19 12:13:11,676 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 12:13:11,677 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 12:13:11,677 INFO repeater task os75nb_beamplots starting 23.5 seconds late 2015-04-19 12:13:13,423 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 12:13:13,423 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 12:13:13,424 INFO repeater task os75nb_beamplots starting 25.3 seconds late 2015-04-19 12:13:15,165 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 12:13:15,165 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 12:13:15,166 INFO repeater task ashtech_hcorrstats starting 27.0 seconds late 2015-04-19 12:13:16,272 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:13:21,278 INFO repeater task os75nb_lastensq starting 24.3 seconds late 2015-04-19 12:13:23,848 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:13:23,849 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:12:47 2015-04-19 12:17:49,228 INFO repeater task ashtech_hcorrstats starting 1.0 seconds late 2015-04-19 12:17:50,334 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:18:00,348 INFO repeater task os75nb_lastensq starting 3.3 seconds late 2015-04-19 12:18:02,921 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:18:02,921 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:17:47 2015-04-19 12:18:12,937 INFO repeater task os75nb_10pings starting 4.8 seconds late 2015-04-19 12:18:13,122 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 12:22:48,470 INFO repeater task ashtech_hcorrstats starting 0.3 seconds late 2015-04-19 12:22:49,568 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:22:59,579 INFO repeater task os75nb_lastensq starting 2.6 seconds late 2015-04-19 12:23:02,149 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:23:02,149 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:22:47 2015-04-19 12:23:32,196 INFO repeater task os75nb_10pings starting 4.0 seconds late 2015-04-19 12:23:32,381 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 12:27:52,726 INFO repeater task os75nb_quick starting 4.6 seconds late 2015-04-19 12:27:59,735 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 12:27:59,736 INFO repeater task os75nb_3dayplots starting 11.6 seconds late 2015-04-19 12:28:12,856 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 12:28:12,856 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2015-04-19 12:28:12,857 INFO repeater task os75nb_tsstats starting 24.7 seconds late 2015-04-19 12:28:13,949 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 12:28:13,950 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 12:28:13,950 INFO repeater task os75nb_beamplots starting 25.8 seconds late 2015-04-19 12:28:15,695 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 12:28:15,695 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 12:28:15,696 INFO repeater task os75nb_beamplots starting 27.5 seconds late 2015-04-19 12:28:17,438 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 12:28:17,438 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 12:28:17,439 INFO repeater task ashtech_hcorrstats starting 29.3 seconds late 2015-04-19 12:28:18,533 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:28:23,539 INFO repeater task os75nb_lastensq starting 26.5 seconds late 2015-04-19 12:28:26,098 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:28:26,098 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:27:47 2015-04-19 12:28:51,155 INFO repeater task os75nb_10pings starting 3.0 seconds late 2015-04-19 12:28:51,335 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 12:32:51,688 INFO repeater task ashtech_hcorrstats starting 3.5 seconds late 2015-04-19 12:32:52,787 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:32:57,793 INFO repeater task os75nb_lastensq starting 0.8 seconds late 2015-04-19 12:33:00,367 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:33:00,368 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:32:47 2015-04-19 12:34:10,472 INFO repeater task os75nb_10pings starting 2.3 seconds late 2015-04-19 12:34:10,653 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 12:37:50,971 INFO repeater task ashtech_hcorrstats starting 2.8 seconds late 2015-04-19 12:37:52,072 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:37:57,078 INFO repeater task os75nb_lastensq starting 0.1 seconds late 2015-04-19 12:37:59,665 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:37:59,665 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:37:47 2015-04-19 12:39:29,790 INFO repeater task os75nb_10pings starting 1.6 seconds late 2015-04-19 12:39:29,972 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 12:42:50,234 INFO repeater task os75nb_quick starting 2.1 seconds late 2015-04-19 12:42:57,288 INFO repeater task os75nb_quick ended after 7.1 seconds 2015-04-19 12:42:57,288 INFO repeater task os75nb_3dayplots starting 9.1 seconds late 2015-04-19 12:43:10,489 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 12:43:10,490 INFO repeater task os75nb_3dayplots ended after 13.2 seconds 2015-04-19 12:43:10,490 INFO repeater task os75nb_tsstats starting 22.3 seconds late 2015-04-19 12:43:11,605 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 12:43:11,605 INFO repeater task os75nb_beamplots starting 23.4 seconds late 2015-04-19 12:43:13,343 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 12:43:13,344 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 12:43:13,344 INFO repeater task os75nb_beamplots starting 25.2 seconds late 2015-04-19 12:43:15,066 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 12:43:15,066 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 12:43:15,066 INFO repeater task ashtech_hcorrstats starting 26.9 seconds late 2015-04-19 12:43:16,160 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:43:21,166 INFO repeater task os75nb_lastensq starting 24.2 seconds late 2015-04-19 12:43:23,747 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:43:23,747 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:42:47 2015-04-19 12:44:48,869 INFO repeater task os75nb_10pings starting 0.7 seconds late 2015-04-19 12:44:49,053 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 12:47:49,278 INFO repeater task ashtech_hcorrstats starting 1.1 seconds late 2015-04-19 12:47:50,378 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:48:00,390 INFO repeater task os75nb_lastensq starting 3.4 seconds late 2015-04-19 12:48:02,956 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:48:02,957 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:47:47 2015-04-19 12:50:13,138 INFO repeater task os75nb_10pings starting 5.0 seconds late 2015-04-19 12:50:13,321 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 12:52:48,516 INFO repeater task ashtech_hcorrstats starting 0.3 seconds late 2015-04-19 12:52:49,618 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:52:59,626 INFO repeater task os75nb_lastensq starting 2.6 seconds late 2015-04-19 12:53:02,199 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:53:02,199 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:52:47 2015-04-19 12:55:32,400 INFO repeater task os75nb_10pings starting 4.2 seconds late 2015-04-19 12:55:32,585 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 12:57:52,779 INFO repeater task os75nb_quick starting 4.6 seconds late 2015-04-19 12:57:59,815 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 12:57:59,815 INFO repeater task os75nb_3dayplots starting 11.6 seconds late 2015-04-19 12:58:12,836 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 12:58:12,836 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2015-04-19 12:58:12,839 INFO repeater task os75nb_tsstats starting 24.7 seconds late 2015-04-19 12:58:13,951 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 12:58:13,951 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 12:58:13,951 INFO repeater task os75nb_beamplots starting 25.8 seconds late 2015-04-19 12:58:15,674 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 12:58:15,674 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 12:58:15,674 INFO repeater task os75nb_beamplots starting 27.5 seconds late 2015-04-19 12:58:17,387 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 12:58:17,387 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 12:58:17,387 INFO repeater task ashtech_hcorrstats starting 29.2 seconds late 2015-04-19 12:58:18,442 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 12:58:23,448 INFO repeater task os75nb_lastensq starting 26.4 seconds late 2015-04-19 12:58:26,042 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 12:58:26,042 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 12:57:47 2015-04-19 13:00:51,245 INFO repeater task os75nb_10pings starting 3.1 seconds late 2015-04-19 13:00:51,439 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 13:02:51,603 INFO repeater task ashtech_hcorrstats starting 3.4 seconds late 2015-04-19 13:02:52,682 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:02:57,688 INFO repeater task os75nb_lastensq starting 0.7 seconds late 2015-04-19 13:03:00,252 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:03:00,253 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:02:47 2015-04-19 13:06:10,527 INFO repeater task os75nb_10pings starting 2.4 seconds late 2015-04-19 13:06:10,712 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 13:07:50,843 INFO repeater task ashtech_hcorrstats starting 2.7 seconds late 2015-04-19 13:07:51,905 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:08:01,918 INFO repeater task os75nb_lastensq starting 4.9 seconds late 2015-04-19 13:08:04,485 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:08:04,486 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:07:47 2015-04-19 13:11:29,771 INFO repeater task os75nb_10pings starting 1.6 seconds late 2015-04-19 13:11:29,958 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 13:12:50,067 INFO repeater task os75nb_quick starting 1.9 seconds late 2015-04-19 13:12:57,111 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 13:12:57,111 INFO repeater task os75nb_3dayplots starting 8.9 seconds late 2015-04-19 13:13:10,160 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 13:13:10,160 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2015-04-19 13:13:10,160 INFO repeater task os75nb_tsstats starting 22.0 seconds late 2015-04-19 13:13:11,262 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 13:13:11,263 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 13:13:11,263 INFO repeater task os75nb_beamplots starting 23.1 seconds late 2015-04-19 13:13:13,000 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 13:13:13,000 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 13:13:13,000 INFO repeater task os75nb_beamplots starting 24.8 seconds late 2015-04-19 13:13:14,744 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 13:13:14,744 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 13:13:14,744 INFO repeater task ashtech_hcorrstats starting 26.6 seconds late 2015-04-19 13:13:15,809 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:13:20,814 INFO repeater task os75nb_lastensq starting 23.8 seconds late 2015-04-19 13:13:23,403 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:13:23,404 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:12:47 2015-04-19 13:16:48,677 INFO repeater task os75nb_10pings starting 0.5 seconds late 2015-04-19 13:16:48,860 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 13:17:48,948 INFO repeater task ashtech_hcorrstats starting 0.8 seconds late 2015-04-19 13:17:50,018 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:18:00,031 INFO repeater task os75nb_lastensq starting 3.0 seconds late 2015-04-19 13:18:02,601 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:18:02,601 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:17:47 2015-04-19 13:22:12,925 INFO repeater task os75nb_10pings starting 4.8 seconds late 2015-04-19 13:22:13,109 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 13:22:53,168 INFO repeater task ashtech_hcorrstats starting 5.0 seconds late 2015-04-19 13:22:54,230 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:22:59,231 INFO repeater task os75nb_lastensq starting 2.2 seconds late 2015-04-19 13:23:01,797 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:23:01,798 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:22:47 2015-04-19 13:27:32,177 INFO repeater task os75nb_10pings starting 4.0 seconds late 2015-04-19 13:27:32,361 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 13:27:52,391 INFO repeater task os75nb_quick starting 4.2 seconds late 2015-04-19 13:27:59,459 INFO repeater task os75nb_quick ended after 7.1 seconds 2015-04-19 13:27:59,460 INFO repeater task os75nb_3dayplots starting 11.3 seconds late 2015-04-19 13:28:12,557 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 13:28:12,557 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2015-04-19 13:28:12,557 INFO repeater task os75nb_tsstats starting 24.4 seconds late 2015-04-19 13:28:13,652 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 13:28:13,652 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 13:28:13,653 INFO repeater task os75nb_beamplots starting 25.5 seconds late 2015-04-19 13:28:15,404 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 13:28:15,405 INFO repeater task os75nb_beamplots ended after 1.8 seconds 2015-04-19 13:28:15,405 INFO repeater task os75nb_beamplots starting 27.2 seconds late 2015-04-19 13:28:17,144 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 13:28:17,144 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 13:28:17,145 INFO repeater task ashtech_hcorrstats starting 29.0 seconds late 2015-04-19 13:28:18,202 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:28:23,208 INFO repeater task os75nb_lastensq starting 26.2 seconds late 2015-04-19 13:28:25,821 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:28:25,821 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:27:47 2015-04-19 13:32:51,236 INFO repeater task os75nb_10pings starting 3.1 seconds late 2015-04-19 13:32:51,419 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 13:32:51,421 INFO repeater task ashtech_hcorrstats starting 3.2 seconds late 2015-04-19 13:32:52,492 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:32:57,498 INFO repeater task os75nb_lastensq starting 0.5 seconds late 2015-04-19 13:33:00,057 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:33:00,057 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:32:47 2015-04-19 13:37:50,415 INFO repeater task ashtech_hcorrstats starting 2.2 seconds late 2015-04-19 13:37:51,493 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:38:01,506 INFO repeater task os75nb_lastensq starting 4.5 seconds late 2015-04-19 13:38:04,090 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:38:04,091 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:37:47 2015-04-19 13:38:09,095 INFO repeater task os75nb_10pings starting 0.9 seconds late 2015-04-19 13:38:09,283 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 13:42:49,631 INFO repeater task os75nb_quick starting 1.5 seconds late 2015-04-19 13:42:56,743 INFO repeater task os75nb_quick ended after 7.1 seconds 2015-04-19 13:42:56,743 INFO repeater task os75nb_3dayplots starting 8.6 seconds late 2015-04-19 13:43:09,819 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 13:43:09,820 INFO repeater task os75nb_3dayplots ended after 13.1 seconds 2015-04-19 13:43:09,820 INFO repeater task os75nb_tsstats starting 21.6 seconds late 2015-04-19 13:43:10,911 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 13:43:10,912 INFO repeater task os75nb_beamplots starting 22.7 seconds late 2015-04-19 13:43:12,650 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 13:43:12,650 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 13:43:12,651 INFO repeater task os75nb_beamplots starting 24.5 seconds late 2015-04-19 13:43:14,370 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 13:43:14,370 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 13:43:14,370 INFO repeater task ashtech_hcorrstats starting 26.2 seconds late 2015-04-19 13:43:15,426 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:43:20,432 INFO repeater task os75nb_lastensq starting 23.4 seconds late 2015-04-19 13:43:23,017 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:43:23,017 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:42:47 2015-04-19 13:43:33,027 INFO repeater task os75nb_10pings starting 4.9 seconds late 2015-04-19 13:43:33,214 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 13:47:48,576 INFO repeater task ashtech_hcorrstats starting 0.4 seconds late 2015-04-19 13:47:49,653 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:47:59,667 INFO repeater task os75nb_lastensq starting 2.7 seconds late 2015-04-19 13:48:02,269 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:48:02,269 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:47:47 2015-04-19 13:48:52,342 INFO repeater task os75nb_10pings starting 4.2 seconds late 2015-04-19 13:48:52,524 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 13:52:52,851 INFO repeater task ashtech_hcorrstats starting 4.7 seconds late 2015-04-19 13:52:53,918 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:52:58,920 INFO repeater task os75nb_lastensq starting 1.9 seconds late 2015-04-19 13:53:01,522 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:53:01,523 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:52:47 2015-04-19 13:54:11,607 INFO repeater task os75nb_10pings starting 3.4 seconds late 2015-04-19 13:54:11,793 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 13:57:52,095 INFO repeater task os75nb_quick starting 3.9 seconds late 2015-04-19 13:57:59,141 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 13:57:59,141 INFO repeater task os75nb_3dayplots starting 11.0 seconds late 2015-04-19 13:58:12,181 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 13:58:12,182 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2015-04-19 13:58:12,183 INFO repeater task os75nb_tsstats starting 24.0 seconds late 2015-04-19 13:58:13,278 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 13:58:13,279 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 13:58:13,279 INFO repeater task os75nb_beamplots starting 25.1 seconds late 2015-04-19 13:58:15,009 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 13:58:15,010 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 13:58:15,010 INFO repeater task os75nb_beamplots starting 26.8 seconds late 2015-04-19 13:58:16,742 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 13:58:16,742 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 13:58:16,743 INFO repeater task ashtech_hcorrstats starting 28.6 seconds late 2015-04-19 13:58:17,816 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 13:58:22,821 INFO repeater task os75nb_lastensq starting 25.8 seconds late 2015-04-19 13:58:25,387 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 13:58:25,387 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 13:57:47 2015-04-19 13:59:30,471 INFO repeater task os75nb_10pings starting 2.3 seconds late 2015-04-19 13:59:30,657 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 14:02:50,924 INFO repeater task ashtech_hcorrstats starting 2.7 seconds late 2015-04-19 14:02:51,986 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 14:03:01,999 INFO repeater task os75nb_lastensq starting 5.0 seconds late 2015-04-19 14:03:04,628 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 14:03:04,628 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 14:02:47 2015-04-19 14:04:49,786 INFO repeater task os75nb_10pings starting 1.6 seconds late 2015-04-19 14:04:49,967 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 14:07:50,237 INFO repeater task ashtech_hcorrstats starting 2.1 seconds late 2015-04-19 14:07:51,300 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 14:08:01,313 INFO repeater task os75nb_lastensq starting 4.3 seconds late 2015-04-19 14:08:03,861 INFO repeater task os75nb_lastensq ended after 2.5 seconds 2015-04-19 14:08:03,862 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 14:07:47 2015-04-19 14:10:09,023 INFO repeater task os75nb_10pings starting 0.9 seconds late 2015-04-19 14:10:09,208 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 14:12:49,438 INFO repeater task os75nb_quick starting 1.3 seconds late 2015-04-19 14:12:56,545 INFO repeater task os75nb_quick ended after 7.1 seconds 2015-04-19 14:12:56,545 INFO repeater task os75nb_3dayplots starting 8.4 seconds late 2015-04-19 14:13:09,533 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) zgrid failed zgrid failed: returning all values masked 2015-04-19 14:13:09,533 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2015-04-19 14:13:09,533 INFO repeater task os75nb_tsstats starting 21.4 seconds late 2015-04-19 14:13:10,637 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 14:13:10,637 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 14:13:10,638 INFO repeater task os75nb_beamplots starting 22.5 seconds late 2015-04-19 14:13:12,385 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 14:13:12,385 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 14:13:12,385 INFO repeater task os75nb_beamplots starting 24.2 seconds late 2015-04-19 14:13:14,119 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 14:13:14,119 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 14:13:14,119 INFO repeater task ashtech_hcorrstats starting 25.9 seconds late 2015-04-19 14:13:15,182 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 14:13:20,188 INFO repeater task os75nb_lastensq starting 23.2 seconds late 2015-04-19 14:13:22,753 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 14:13:22,754 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 14:12:47 2015-04-19 14:15:32,952 INFO repeater task os75nb_10pings starting 4.8 seconds late 2015-04-19 14:15:33,135 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 14:17:48,337 INFO repeater task ashtech_hcorrstats starting 0.2 seconds late 2015-04-19 14:17:49,398 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 14:17:59,411 INFO repeater task os75nb_lastensq starting 2.4 seconds late 2015-04-19 14:18:01,977 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 14:18:01,978 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 14:17:47 2015-04-19 14:20:52,226 INFO repeater task os75nb_10pings starting 4.1 seconds late 2015-04-19 14:20:52,411 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 14:22:52,566 INFO repeater task ashtech_hcorrstats starting 4.4 seconds late 2015-04-19 14:22:53,631 INFO repeater task ashtech_hcorrstats ended after 1.1 seconds 2015-04-19 14:22:58,637 INFO repeater task os75nb_lastensq starting 1.6 seconds late 2015-04-19 14:23:01,211 INFO repeater task os75nb_lastensq ended after 2.6 seconds 2015-04-19 14:23:01,211 INFO repeater task os75nb_lastensq: Time from /home/data/CalCOFI1504/proc/os75nb/load/wrote_ens_time: 2015/04/19 14:22:47 2015-04-19 14:26:11,491 INFO repeater task os75nb_10pings starting 3.3 seconds late 2015-04-19 14:26:11,673 INFO repeater task os75nb_10pings ended after 0.2 seconds 2015-04-19 14:27:51,804 INFO repeater task os75nb_quick starting 3.6 seconds late 2015-04-19 14:27:58,849 INFO repeater task os75nb_quick ended after 7.0 seconds 2015-04-19 14:27:58,850 INFO repeater task os75nb_3dayplots starting 10.7 seconds late 2015-04-19 14:28:11,894 INFO repeater task os75nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2015-04-19 14:28:11,895 INFO repeater task os75nb_3dayplots ended after 13.0 seconds 2015-04-19 14:28:11,895 INFO repeater task os75nb_tsstats starting 23.7 seconds late 2015-04-19 14:28:12,995 INFO repeater task os75nb_tsstats output: /usr/lib/python2.7/dist-packages/numpy/ma/core.py:1073: RuntimeWarning: overflow encountered in true_divide result = self.f(da, db, *args, **kwargs) 2015-04-19 14:28:12,995 INFO repeater task os75nb_tsstats ended after 1.1 seconds 2015-04-19 14:28:12,995 INFO repeater task os75nb_beamplots starting 24.8 seconds late 2015-04-19 14:28:14,712 INFO repeater task os75nb_beamplots output: Starting procsetup scriptpath is /home/currents/programs/uhdas/scripts CURRENTS_REPO_PATH not found; assuming in-place installation programs_path is /home/currents/programs 2015-04-19 14:28:14,712 INFO repeater task os75nb_beamplots ended after 1.7 seconds 2015-04-19 14:28:14,713 INFO repeater task os75nb_beamplots starting 26.5 seconds late