Skip to content

Subtract extra leap second from GCD file #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
gh999ic wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Subtract extra leap second from GCD file #32

gh999ic wants to merge 6 commits into from

Conversation

@gh999ic
Copy link
Collaborator

@gh999ic gh999ic commented Oct 26, 2025 •
edited
Loading

GoodRunStartTime and GoodRunEndTime in IC86-2016 GCD files from calendar year 2017 are 1 second greater than i3Live.
The scripts in this PR will subtract the extra leap second from GoodRunStartTime and GoodRunEndTime in IC86-2016 GCD files from calendar year 2017.
This extra leap second was erroneously added in the original GCD files,
when it was already accounted for in i3Live.
This affects runs 129004 (2017-01-01 first run of calendar year) to 129519 (2017-05-18 last run of IC86-2016).

The GoodRunStartTime and GoodRunEndTime in the GCD file are used by offline processing to exclude any events outside their time range.

@briedel
Copy link
Contributor

briedel commented Oct 26, 2025 via email

@gh999ic
Copy link
Collaborator Author

gh999ic commented Oct 27, 2025 •
edited
Loading

I just learned about the nist leap second file recently from Tim (perhaps @arcor can comment).
I looked at the original log file from executing the GCD generation script for L2 in 2017.
The script first prints the good run start and end time obtained from the i3Live DB,
and secondly prints the good run start and end time that are written into the GCD file.
The log printout from the last run 129002 of calendar year 2016 shows no change
in these two sets of good run start and end time:

20251026Sun23:38:02 i3filter@cobalt-15  203% lf /data/exp/IceCube/2016/filtered/level2/OfflinePreChecks/run_logs/logs/1231/Run00129002_55_290.out 
-rw-r--r-- 1 i3filter i3filter 39243 2017-01-13 08:27:22.000000000 -0600 /data/exp/IceCube/2016/filtered/level2/OfflinePreChecks/run_logs/logs/1231/Run00129002_55_290.out
20251026Sun23:38:15 i3filter@cobalt-15  204% head /data/exp/IceCube/2016/filtered/level2/OfflinePreChecks/run_logs/logs/1231/Run00129002_55_290.out 
Processing run 129002 with ProcessingVersion:55 SnapshotId:290
ScratchSpace:  /scratch/condor/dir_21504
[{u'run_id': 129002, u'TemplateGCDCheck': None, u'good_it': 1, u'snapshot_id': 290, u'validated': 0, u'GCDCheck': 0, u'ActiveInIceDOMs': None, u'good_tstart_frac': 4330449116, u'ss_ref': 7160, u'good_tstop_frac': 5218010433, u'comments': '', u'production_version': 55, u'good_tstop': datetime.datetime(2016, 12, 31, 23, 59, 3), u'reason_it': '', u'BadDOMsCheck': 0, u'ActiveDOMs': None, u'good_tstart': datetime.datetime(2016, 12, 31, 23, 1, 19), u'good_i3': 1, u'ActiveStrings': None, u'submitted': 0, u'PoleGCDCheck': None, u'reason_i3': ''}]

Generating GCD file for run 129002
129002 /data/exp/IceCube/2016/filtered/PFFilt/1231/PFFilt_PhysicsFiltering_Run00129002_Subrun00000000_00000000.tar.bz2 /data/exp/IceCube/2016/filtered/level2/OfflinePreChecks/DataFiles/1231/Level2_IC86.2016_data_Run00129002_55_290_GCD.i3.gz 55 290 2016-12-31 23:01:19.433,044,911,6 UTC 2016-12-31 23:59:03.521,801,043,3 UTC

==== Completed GCD generation attempt for run 129002

Whereas those runs in calendar year 2017 (eg. first run 129004) added one second to the latter set of good run start and end time (i.e., the good run start and end time that are written into the GCD files):

20251026Sun23:36:15 i3filter@cobalt-15  199% lf /data/exp/IceCube/2017/filtered/level2/OfflinePreChecks/run_logs/logs/0101/Run00129004_55_290.out 
-rw-r--r-- 1 i3filter IceC-filt 37823 2017-01-17 13:30:21.000000000 -0600 /data/exp/IceCube/2017/filtered/level2/OfflinePreChecks/run_logs/logs/0101/Run00129004_55_290.out
20251026Sun23:36:36 i3filter@cobalt-15  200% head /data/exp/IceCube/2017/filtered/level2/OfflinePreChecks/run_logs/logs/0101/Run00129004_55_290.out 
Processing run 129004 with ProcessingVersion:55 SnapshotId:290
ScratchSpace:  /scratch/condor/dir_12825
[{u'run_id': 129004, u'TemplateGCDCheck': None, u'good_it': 1, u'snapshot_id': 290, u'validated': 0, u'GCDCheck': 0, u'ActiveInIceDOMs': None, u'good_tstart_frac': 3136210276, u'ss_ref': 7162, u'good_tstop_frac': 6477005069, u'comments': '', u'production_version': 55, u'good_tstop': datetime.datetime(2017, 1, 1, 8, 7, 31), u'reason_it': '', u'BadDOMsCheck': 0, u'ActiveDOMs': None, u'good_tstart': datetime.datetime(2017, 1, 1, 0, 8, 5), u'good_i3': 1, u'ActiveStrings': None, u'submitted': 1, u'PoleGCDCheck': None, u'reason_i3': ''}]

Generating GCD file for run 129004
129004 /data/exp/IceCube/2017/filtered/PFFilt/0101/PFFilt_PhysicsFiltering_Run00129004_Subrun00000000_00000000.tar.bz2 /data/exp/IceCube/2017/filtered/level2/OfflinePreChecks/DataFiles/0101/Level2_IC86.2016_data_Run00129004_55_290_GCD.i3.gz 55 290 2017-01-01 00:08:06.313,621,027,6 UTC 2017-01-01 08:07:32.647,700,506,9 UTC

==== Completed GCD generation attempt for run 129004

The leap-second file that was used appears to be the following.
Seems like that extra leap second was added only for calendar year 2017.

20251026Sun23:31:23 i3filter@cobalt-15  194% lf /data/user/i3filter/IC86_OfflineProcessing/OfflineSubmitScripts_2016/tmp/leap-seconds.list 
-rw-rw-r-- 1 i3filter i3filter 10405 2016-08-17 09:57:14.000000000 -0500 /data/user/i3filter/IC86_OfflineProcessing/OfflineSubmitScripts_2016/tmp/leap-seconds.list
20251026Sun23:34:38 i3filter@cobalt-15  195% cat /data/user/i3filter/IC86_OfflineProcessing/OfflineSubmitScripts_2016/tmp/leap-seconds.list 
#
#       In the following text, the symbol '#' introduces
#       a comment, which continues from that symbol until
#       the end of the line. A plain comment line has a
#       whitespace character following the comment indicator.
#       There are also special comment lines defined below.
#       A special comment will always have a non-whitespace
#       character in column 2.
#
#       A blank line should be ignored.
#
#       The following table shows the corrections that must
#       be applied to compute International Atomic Time (TAI)
#       from the Coordinated Universal Time (UTC) values that
#       are transmitted by almost all time services.
#
#       The first column shows an epoch as a number of seconds
#       since 1 January 1900, 00:00:00 (1900.0 is also used to
#       indicate the same epoch.) Both of these time stamp formats
#       ignore the complexities of the time scales that were
#       used before the current definition of UTC at the start
#       of 1972. (See note 3 below.)
#       The second column shows the number of seconds that
#       must be added to UTC to compute TAI for any timestamp
#       at or after that epoch. The value on each line is
#       valid from the indicated initial instant until the
#       epoch given on the next one or indefinitely into the
#       future if there is no next line.
#       (The comment on each line shows the representation of
#       the corresponding initial epoch in the usual
#       day-month-year format. The epoch always begins at
#       00:00:00 UTC on the indicated day. See Note 5 below.)
#
#       Important notes:
#
#       1. Coordinated Universal Time (UTC) is often referred to
#       as Greenwich Mean Time (GMT). The GMT time scale is no
#       longer used, and the use of GMT to designate UTC is
#       discouraged.
#
#       2. The UTC time scale is realized by many national
#       laboratories and timing centers. Each laboratory
#       identifies its realization with its name: Thus
#       UTC(NIST), UTC(USNO), etc. The differences among
#       these different realizations are typically on the
#       order of a few nanoseconds (i.e., 0.000 000 00x s)
#       and can be ignored for many purposes. These differences
#       are tabulated in Circular T, which is published monthly
#       by the International Bureau of Weights and Measures
#       (BIPM). See www.bipm.org for more information.
#
#       3. The current definition of the relationship between UTC
#       and TAI dates from 1 January 1972. A number of different
#       time scales were in use before that epoch, and it can be
#       quite difficult to compute precise timestamps and time
#       intervals in those "prehistoric" days. For more information,
#       consult:
#
#               The Explanatory Supplement to the Astronomical
#               Ephemeris.
#       or
#               Terry Quinn, "The BIPM and the Accurate Measurement
#               of Time," Proc. of the IEEE, Vol. 79, pp. 894-905,
#               July, 1991.
#
#       4. The decision to insert a leap second into UTC is currently
#       the responsibility of the International Earth Rotation and
#       Reference Systems Service. (The name was changed from the
#       International Earth Rotation Service, but the acronym IERS
#       is still used.)
#
#       Leap seconds are announced by the IERS in its Bulletin C.
#
#       See www.iers.org for more details.
#
#       Every national laboratory and timing center uses the
#       data from the BIPM and the IERS to construct UTC(lab),
#       their local realization of UTC.
#
#       Although the definition also includes the possibility
#       of dropping seconds ("negative" leap seconds), this has
#       never been done and is unlikely to be necessary in the
#       foreseeable future.
#
#       5. If your system keeps time as the number of seconds since
#       some epoch (e.g., NTP timestamps), then the algorithm for
#       assigning a UTC time stamp to an event that happens during a positive
#       leap second is not well defined. The official name of that leap
#       second is 23:59:60, but there is no way of representing that time
#       in these systems.
#       Many systems of this type effectively stop the system clock for
#       one second during the leap second and use a time that is equivalent
#       to 23:59:59 UTC twice. For these systems, the corresponding TAI
#       timestamp would be obtained by advancing to the next entry in the
#       following table when the time equivalent to 23:59:59 UTC
#       is used for the second time. Thus the leap second which
#       occurred on 30 June 1972 at 23:59:59 UTC would have TAI
#       timestamps computed as follows:
#
#       ...
#       30 June 1972 23:59:59 (2287785599, first time): TAI= UTC + 10 seconds
#       30 June 1972 23:59:60 (2287785599,second time): TAI= UTC + 11 seconds
#       1  July 1972 00:00:00 (2287785600)              TAI= UTC + 11 seconds
#       ...
#
#       If your system realizes the leap second by repeating 00:00:00 UTC twice
#       (this is possible but not usual), then the advance to the next entry
#       in the table must occur the second time that a time equivalent to
#       00:00:00 UTC is used. Thus, using the same example as above:
#
#       ...
#       30 June 1972 23:59:59 (2287785599):             TAI= UTC + 10 seconds
#       30 June 1972 23:59:60 (2287785600, first time): TAI= UTC + 10 seconds
#       1  July 1972 00:00:00 (2287785600,second time): TAI= UTC + 11 seconds
#       ...
#
#       in both cases the use of timestamps based on TAI produces a smooth
#       time scale with no discontinuity in the time interval. However,
#       although the long-term behavior of the time scale is correct in both
#       methods, the second method is technically not correct because it adds
#       the extra second to the wrong day.
#
#       This complexity would not be needed for negative leap seconds (if they
#       are ever used). The UTC time would skip 23:59:59 and advance from
#       23:59:58 to 00:00:00 in that case. The TAI offset would decrease by
#       1 second at the same instant. This is a much easier situation to deal
#       with, since the difficulty of unambiguously representing the epoch
#       during the leap second does not arise.
#
#       Some systems implement leap seconds by amortizing the leap second
#       over the last few minutes of the day. The frequency of the local
#       clock is decreased (or increased) to realize the positive (or
#       negative) leap second. This method removes the time step described
#       above. Although the long-term behavior of the time scale is correct
#       in this case, this method introduces an error during the adjustment
#       period both in time and in frequency with respect to the official
#       definition of UTC.
#
#       Questions or comments to:
#               Judah Levine
#               Time and Frequency Division
#               NIST
#               Boulder, Colorado
#               Judah.Levine@nist.gov
#
#       Last Update of leap second values:   8 July 2016
#
#       The following line shows this last update date in NTP timestamp
#       format. This is the date on which the most recent change to
#       the leap second data was added to the file. This line can
#       be identified by the unique pair of characters in the first two
#       columns as shown below.
#
#$       3676924800
#
#       The NTP timestamps are in units of seconds since the NTP epoch,
#       which is 1 January 1900, 00:00:00. The Modified Julian Day number
#       corresponding to the NTP time stamp, X, can be computed as
#
#       X/86400 + 15020
#
#       where the first term converts seconds to days and the second
#       term adds the MJD corresponding to the time origin defined above.
#       The integer portion of the result is the integer MJD for that
#       day, and any remainder is the time of day, expressed as the
#       fraction of the day since 0 hours UTC. The conversion from day
#       fraction to seconds or to hours, minutes, and seconds may involve
#       rounding or truncation, depending on the method used in the
#       computation.
#
#       The data in this file will be updated periodically as new leap
#       seconds are announced. In addition to being entered on the line
#       above, the update time (in NTP format) will be added to the basic
#       file name leap-seconds to form the name leap-seconds.<NTP TIME>.
#       In addition, the generic name leap-seconds.list will always point to
#       the most recent version of the file.
#
#       This update procedure will be performed only when a new leap second
#       is announced.
#
#       The following entry specifies the expiration date of the data
#       in this file in units of seconds since the origin at the instant
#       1 January 1900, 00:00:00. This expiration date will be changed
#       at least twice per year whether or not a new leap second is
#       announced. These semi-annual changes will be made no later
#       than 1 June and 1 December of each year to indicate what
#       action (if any) is to be taken on 30 June and 31 December,
#       respectively. (These are the customary effective dates for new
#       leap seconds.) This expiration date will be identified by a
#       unique pair of characters in columns 1 and 2 as shown below.
#       In the unlikely event that a leap second is announced with an
#       effective date other than 30 June or 31 December, then this
#       file will be edited to include that leap second as soon as it is
#       announced or at least one month before the effective date
#       (whichever is later).
#       If an announcement by the IERS specifies that no leap second is
#       scheduled, then only the expiration date of the file will
#       be advanced to show that the information in the file is still
#       current -- the update time stamp, the data and the name of the file
#       will not change.
#
#       Updated through IERS Bulletin C52
#       File expires on:  28 June 2017
#
#@      3707596800
#
2272060800      10      # 1 Jan 1972
2287785600      11      # 1 Jul 1972
2303683200      12      # 1 Jan 1973
2335219200      13      # 1 Jan 1974
2366755200      14      # 1 Jan 1975
2398291200      15      # 1 Jan 1976
2429913600      16      # 1 Jan 1977
2461449600      17      # 1 Jan 1978
2492985600      18      # 1 Jan 1979
2524521600      19      # 1 Jan 1980
2571782400      20      # 1 Jul 1981
2603318400      21      # 1 Jul 1982
2634854400      22      # 1 Jul 1983
2698012800      23      # 1 Jul 1985
2776982400      24      # 1 Jan 1988
2840140800      25      # 1 Jan 1990
2871676800      26      # 1 Jan 1991
2918937600      27      # 1 Jul 1992
2950473600      28      # 1 Jul 1993
2982009600      29      # 1 Jul 1994
3029443200      30      # 1 Jan 1996
3076704000      31      # 1 Jul 1997
3124137600      32      # 1 Jan 1999
3345062400      33      # 1 Jan 2006
3439756800      34      # 1 Jan 2009
3550089600      35      # 1 Jul 2012
3644697600      36      # 1 Jul 2015
3692217600      37      # 1 Jan 2017
#
#       the following special comment contains the
#       hash value of the data in this file computed
#       use the secure hash algorithm as specified
#       by FIPS 180-1. See the files in ~/pub/sha for
#       the details of how this hash value is
#       computed. Note that the hash computation
#       ignores comments and whitespace characters
#       in data lines. It includes the NTP values
#       of both the last modification time and the
#       expiration time of the file, but not the
#       white space on those lines.
#       the hash line is also ignored in the
#       computation.
#
#h      dacf2c42 2c4765d6 3c797af8 2cf630eb 699c8c67

@gh999ic gh999ic marked this pull request as ready for review October 27, 2025 15:42
@gh999ic gh999ic mentioned this pull request Oct 27, 2025
Open
@arcor
Copy link

arcor commented Oct 27, 2025

I think the problem may have originated from

/data/user/i3filter/IC86_OfflineProcessing/OfflineSubmitScripts_2016/libs/times.py

which converts a UTC time to daq ticks.

looks like it misinterprets
3692217600 37 # 1 Jan 2017

to mean a leap second happened on Jan, 1 2017 instead of the intended meaning that the prior second (12/31/2016 24:59:60) was a leap second.

This defect was latent (or the script was not in use) for other contemporary leap seconds which were scheduled mid-year.

3550089600      35      # 1 Jul 2012
3644697600      36      # 1 Jul 2015
3692217600      37      # 1 Jan 2017

So when this script was in use for good start times landing in 2017 it mistakenly added an extra 1E10 daq ticks to them during the conversion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@gh999ic @briedel @arcor @kathrawlins