Usage

[1]:

from importlib import reload  ## only for developping
from pathlib import Path
import numpy as np

import matplotlib.pyplot as plt
plt.style.use('tableau-colorblind10')

from obspy.clients.filesystem.sds import Client
from obspy.core import UTCDateTime as UTC

from data_quality_control import sds_db, dqclogging, util, analysis, base

Matplotlib is building the font cache; this may take a moment.

Logger

In a script or notebook, the logger can be set once at the beginning. The routine dqclogging.configure_handlers allows to set to different handler, one writing message to the console and optionally one that writes to a file. It is possible to set different levels for the two.

[2]:

reload(dqclogging)
loglevel_console = "INFO"
loglevel_file = None # No log file

dqclogging.configure_handlers(loglevel_console, loglevel_file)

Define parameters

[3]:

# NSLC
nslc_code = "GR.BFO..BHZ"

# Processing parameters
overlap = 60
fmin, fmax = (4, 14)
nperseg = 2048
winlen_in_s = 3600
#proclen = 24*3600
sampling_rate = 20

# Data sources
sds_root = Path('../sample_sds/').absolute()
inventory_routing_type = "eida-routing"

# Output configuration
outdir = Path('output/usage_demo')
fileunit = "year" # period to store in 1 file

Create output directory:

[4]:

outdir.mkdir(parents=True, exist_ok=True)

You can use the sds-Client directly to check the content of the database. Note though, that this can take some time if your database is large.

[5]:

sdsclient = Client(str(sds_root))
sdsclient.get_all_nslc()

[5]:

[('GR', 'BFO', '', 'BHZ')]

Process raw data

Since we have an sds-database, we use the sds_db module to extract amplitudes and power spectral densities (PSD) from the raw seismic data.

[6]:

reload(sds_db)  # only for developping
processor = sds_db.SDSProcessor(
        nslc_code,
        inventory_routing_type,
        sds_root,
        outdir=outdir,
        fileunit=fileunit,
        # Processing parameters
        overlap=overlap, nperseg=nperseg,
        winlen_seconds=winlen_in_s,
        #proclen_seconds=proclen,
        amplitude_frequencies=(fmin, fmax),
        sampling_rate=sampling_rate)

print(processor)
#processor.logger.setLevel("INFO")

overlap different from default
amplitude_frequencies different from default
nperseg different from default
winlen_seconds different from default
using default for proclen_seconds
sampling_rate different from default
GR.BFO..BHZ
Data is sent to output/usage_demo
Data client:
sds_root=/home/docs/checkouts/readthedocs.org/user_builds/lonesam/checkouts/latest/docs/source/../sample_sds; sds_type=D; format=MSEED; fileborder_seconds=30; fileborder_samples=5000
Inventory client _user_agent=ObsPy/1.2.2 (Linux-5.19.0-1028-aws-x86_64-with-glibc2.31, Python 3.9.7); _timeout=120; _debug=False; _BaseRoutingClient__include_providers=[]; _BaseRoutingClient__exclude_providers=[]; credentials={}; _url=http://www.orfeus-eu.org/eidaws/routing/1
Processing settings:
overlap = 60
amplitude_frequencies = (4, 14)
nperseg = 2048
winlen_seconds = 3600
proclen_seconds = 86400
sampling_rate = 20
Preprocessing of seismic data: process_stream
Windows per proclen: 24

[7]:

startdate = UTC("2020-12-20")
enddate = UTC("2021-01-15")

[8]:

%%time
#it -n1 -r7
processor.process(startdate, enddate, force_new_file=True)

24-02-09 13:25:37 - data_quality_control.base.ProcessedDataFileManager - INFO - Processed data stored per year in output/usage_demo as {outdir}/{network}.{station}.{location}.{channel}_{year:04d}.hdf5
24-02-09 13:25:37 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2020-12-20T00:00:00.000000Z - 2021-01-01T00:00:00.000000Z
/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/io/stationxml/core.py:96: UserWarning: The StationXML file has version 1.2, ObsPy can read versions (1.0, 1.1). Proceed with caution.
  warnings.warn("The StationXML file has version %s, ObsPy can "
24-02-09 13:25:39 - data_quality_control.base.NSCProcessor - INFO - Processing GR.BFO..BHZ
24-02-09 13:25:39 - data_quality_control.base.NSCProcessor - ERROR - No data for 2020-12-19T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:39 - data_quality_control.base.NSCProcessor - ERROR - No data for 2020-12-20T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:39 - data_quality_control.base.NSCProcessor - ERROR - No data for 2020-12-21T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:39 - data_quality_control.base.NSCProcessor - ERROR - No data for 2020-12-22T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:39 - data_quality_control.util - INFO - Found nans in GR.BFO..BHZ | 2020-12-23T23:59:00.019538Z - 2020-12-25T00:00:59.969538Z | 20.0 Hz, 1730400 samples
/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/signal/filter.py:67: UserWarning: Selected high corner frequency (14) of bandpass is at or above Nyquist (10.0). Applying a high-pass instead.
  warnings.warn(msg)
/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/numpy/lib/nanfunctions.py:1395: RuntimeWarning: All-NaN slice encountered
  result = np.apply_along_axis(_nanquantile_1d, axis, a, q,
24-02-09 13:25:39 - data_quality_control.util - INFO - Found nans in GR.BFO..BHZ | 2020-12-24T23:59:00.019538Z - 2020-12-26T00:00:59.969538Z | 20.0 Hz, 1730400 samples
24-02-09 13:25:41 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2020-12-25T01:00:00.000000Z - 2021-01-01T00:00:00.000000Z
24-02-09 13:25:41 - data_quality_control.base.ProcessedDataFileManager - INFO - Creating output file output/usage_demo/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:41 - data_quality_control.base.ProcessedDataFileManager - INFO - Starttime=2020-01-01T00:00:00.000000Z, endtime=2021-01-01T00:00:00.000000Z, n_windows=8784
24-02-09 13:25:41 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2021-01-01T00:00:00.000000Z - 2021-01-16T00:00:00.000000Z

<HDF5 file "GR.BFO..BHZ_2020.hdf5" (mode r+)>

/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/io/stationxml/core.py:96: UserWarning: The StationXML file has version 1.2, ObsPy can read versions (1.0, 1.1). Proceed with caution.
  warnings.warn("The StationXML file has version %s, ObsPy can "
24-02-09 13:25:41 - data_quality_control.base.NSCProcessor - INFO - Processing GR.BFO..BHZ
/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/signal/filter.py:67: UserWarning: Selected high corner frequency (14) of bandpass is at or above Nyquist (10.0). Applying a high-pass instead.
  warnings.warn(msg)
24-02-09 13:25:43 - data_quality_control.util - INFO - Found nans in GR.BFO..BHZ | 2021-01-08T23:59:00.019538Z - 2021-01-10T00:00:59.969538Z | 20.0 Hz, 1730400 samples
/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/numpy/lib/nanfunctions.py:1395: RuntimeWarning: All-NaN slice encountered
  result = np.apply_along_axis(_nanquantile_1d, axis, a, q,
24-02-09 13:25:43 - data_quality_control.util - INFO - Found nans in GR.BFO..BHZ | 2021-01-09T23:59:00.019538Z - 2021-01-11T00:00:59.969538Z | 20.0 Hz, 1730400 samples
24-02-09 13:25:43 - data_quality_control.base.NSCProcessor - ERROR - No data for 2021-01-10T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:43 - data_quality_control.base.NSCProcessor - ERROR - No data for 2021-01-11T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:43 - data_quality_control.base.NSCProcessor - ERROR - No data for 2021-01-12T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:43 - data_quality_control.base.NSCProcessor - ERROR - No data for 2021-01-13T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:43 - data_quality_control.base.NSCProcessor - ERROR - No data for 2021-01-14T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:43 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2021-01-01T00:00:00.000000Z - 2021-01-09T23:00:00.000000Z
24-02-09 13:25:43 - data_quality_control.base.ProcessedDataFileManager - INFO - Creating output file output/usage_demo/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:43 - data_quality_control.base.ProcessedDataFileManager - INFO - Starttime=2021-01-01T00:00:00.000000Z, endtime=2022-01-01T00:00:00.000000Z, n_windows=8760
24-02-09 13:25:43 - data_quality_control.sds_db.SDSProcessor - INFO - Finished. Took 0:00:05.627335 h

<HDF5 file "GR.BFO..BHZ_2021.hdf5" (mode r+)>
CPU times: user 2.65 s, sys: 562 ms, total: 3.21 s
Wall time: 5.63 s

If we change the fileunit to "month", we get different filenames, indicating also the month.

[9]:

reload(sds_db)  # only for developping
processor = sds_db.SDSProcessor(
        nslc_code,
        inventory_routing_type,
        sds_root,
        outdir=outdir,
        fileunit="month",
        # Processing parameters
        overlap=overlap, nperseg=nperseg,
        winlen_seconds=winlen_in_s,
        #proclen_seconds=proclen,
        amplitude_frequencies=(fmin, fmax))

print(processor)

overlap different from default
amplitude_frequencies different from default
nperseg different from default
winlen_seconds different from default
using default for proclen_seconds
using default for sampling_rate
GR.BFO..BHZ
Data is sent to output/usage_demo
Data client:
sds_root=/home/docs/checkouts/readthedocs.org/user_builds/lonesam/checkouts/latest/docs/source/../sample_sds; sds_type=D; format=MSEED; fileborder_seconds=30; fileborder_samples=5000
Inventory client _user_agent=ObsPy/1.2.2 (Linux-5.19.0-1028-aws-x86_64-with-glibc2.31, Python 3.9.7); _timeout=120; _debug=False; _BaseRoutingClient__include_providers=[]; _BaseRoutingClient__exclude_providers=[]; credentials={}; _url=http://www.orfeus-eu.org/eidaws/routing/1
Processing settings:
overlap = 60
amplitude_frequencies = (4, 14)
nperseg = 2048
winlen_seconds = 3600
proclen_seconds = 86400
sampling_rate = 20
Preprocessing of seismic data: process_stream
Windows per proclen: 24

[10]:

startdate = UTC("2020-12-20")
enddate = UTC("2021-01-15")

[11]:

%%time
#it -n1 -r7
processor.process(startdate, enddate, force_new_file=True)

24-02-09 13:25:43 - data_quality_control.base.ProcessedDataFileManager - INFO - Processed data stored per month in output/usage_demo as {outdir}/{network}.{station}.{location}.{channel}_{year:04d}-{month:02d}.hdf5
24-02-09 13:25:43 - data_quality_control.util - INFO -
Processing 2020-12-20T00:00:00.000000Z - 2020-12-31T00:00:00.000000Z
24-02-09 13:25:43 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2020-12-20T00:00:00.000000Z - 2021-01-01T00:00:00.000000Z
/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/io/stationxml/core.py:96: UserWarning: The StationXML file has version 1.2, ObsPy can read versions (1.0, 1.1). Proceed with caution.
  warnings.warn("The StationXML file has version %s, ObsPy can "
24-02-09 13:25:44 - data_quality_control.base.NSCProcessor - INFO - Processing GR.BFO..BHZ
24-02-09 13:25:44 - data_quality_control.base.NSCProcessor - ERROR - No data for 2020-12-19T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:44 - data_quality_control.base.NSCProcessor - ERROR - No data for 2020-12-20T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:44 - data_quality_control.base.NSCProcessor - ERROR - No data for 2020-12-21T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:44 - data_quality_control.base.NSCProcessor - ERROR - No data for 2020-12-22T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:44 - data_quality_control.util - INFO - Found nans in GR.BFO..BHZ | 2020-12-23T23:59:00.019538Z - 2020-12-25T00:00:59.969538Z | 20.0 Hz, 1730400 samples
/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/signal/filter.py:67: UserWarning: Selected high corner frequency (14) of bandpass is at or above Nyquist (10.0). Applying a high-pass instead.
  warnings.warn(msg)
/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/numpy/lib/nanfunctions.py:1395: RuntimeWarning: All-NaN slice encountered
  result = np.apply_along_axis(_nanquantile_1d, axis, a, q,
24-02-09 13:25:44 - data_quality_control.util - INFO - Found nans in GR.BFO..BHZ | 2020-12-24T23:59:00.019538Z - 2020-12-26T00:00:59.969538Z | 20.0 Hz, 1730400 samples
24-02-09 13:25:45 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2020-12-25T01:00:00.000000Z - 2021-01-01T00:00:00.000000Z
24-02-09 13:25:45 - data_quality_control.base.ProcessedDataFileManager - INFO - Creating output file output/usage_demo/GR.BFO..BHZ_2020-12.hdf5
24-02-09 13:25:45 - data_quality_control.base.ProcessedDataFileManager - INFO - Starttime=2020-12-01T00:00:00.000000Z, endtime=2021-01-01T00:00:00.000000Z, n_windows=744
24-02-09 13:25:45 - data_quality_control.util - INFO -
Processing 2021-01-01T00:00:00.000000Z - 2021-01-15T00:00:00.000000Z
24-02-09 13:25:45 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2021-01-01T00:00:00.000000Z - 2021-01-16T00:00:00.000000Z

<HDF5 file "GR.BFO..BHZ_2020-12.hdf5" (mode r+)>

/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/io/stationxml/core.py:96: UserWarning: The StationXML file has version 1.2, ObsPy can read versions (1.0, 1.1). Proceed with caution.
  warnings.warn("The StationXML file has version %s, ObsPy can "
24-02-09 13:25:46 - data_quality_control.base.NSCProcessor - INFO - Processing GR.BFO..BHZ
/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/signal/filter.py:67: UserWarning: Selected high corner frequency (14) of bandpass is at or above Nyquist (10.0). Applying a high-pass instead.
  warnings.warn(msg)
24-02-09 13:25:47 - data_quality_control.util - INFO - Found nans in GR.BFO..BHZ | 2021-01-08T23:59:00.019538Z - 2021-01-10T00:00:59.969538Z | 20.0 Hz, 1730400 samples
/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/numpy/lib/nanfunctions.py:1395: RuntimeWarning: All-NaN slice encountered
  result = np.apply_along_axis(_nanquantile_1d, axis, a, q,
24-02-09 13:25:47 - data_quality_control.util - INFO - Found nans in GR.BFO..BHZ | 2021-01-09T23:59:00.019538Z - 2021-01-11T00:00:59.969538Z | 20.0 Hz, 1730400 samples
24-02-09 13:25:48 - data_quality_control.base.NSCProcessor - ERROR - No data for 2021-01-10T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:48 - data_quality_control.base.NSCProcessor - ERROR - No data for 2021-01-11T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:48 - data_quality_control.base.NSCProcessor - ERROR - No data for 2021-01-12T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:48 - data_quality_control.base.NSCProcessor - ERROR - No data for 2021-01-13T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:48 - data_quality_control.base.NSCProcessor - ERROR - No data for 2021-01-14T23:59:00.000000Z
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/base.py", line 534, in process
    tr = preprocessing(st, inv, starttime, endtime,
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/data_quality_control/util.py", line 124, in process_stream
    tr = st[0]
  File "/home/docs/checkouts/readthedocs.org/user_builds/lonesam/conda/latest/lib/python3.9/site-packages/obspy/core/stream.py", line 649, in __getitem__
    return self.traces.__getitem__(index)
IndexError: list index out of range
24-02-09 13:25:48 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2021-01-01T00:00:00.000000Z - 2021-01-09T23:00:00.000000Z
24-02-09 13:25:48 - data_quality_control.base.ProcessedDataFileManager - INFO - Creating output file output/usage_demo/GR.BFO..BHZ_2021-01.hdf5
24-02-09 13:25:48 - data_quality_control.base.ProcessedDataFileManager - INFO - Starttime=2021-01-01T00:00:00.000000Z, endtime=2021-02-01T00:00:00.000000Z, n_windows=744
24-02-09 13:25:48 - data_quality_control.sds_db.SDSProcessor - INFO - Finished. Took 0:00:04.483452 h

<HDF5 file "GR.BFO..BHZ_2021-01.hdf5" (mode r+)>
CPU times: user 2.46 s, sys: 550 ms, total: 3.01 s
Wall time: 4.48 s

filunit="month" produces output files with ending YYYY-MM.hdf5. Note that these files are only about 1/12 of the size of the yearly files, indicating that they cover only one month rather than 1 year of data.

[12]:

%ls -lh $outdir

total 75M
-rw-r--r-- 1 docs docs 3.0M Feb  9 13:25 GR.BFO..BHZ_2020-12.hdf5
-rw-r--r-- 1 docs docs  35M Feb  9 13:25 GR.BFO..BHZ_2020.hdf5
-rw-r--r-- 1 docs docs 3.0M Feb  9 13:25 GR.BFO..BHZ_2021-01.hdf5
-rw-r--r-- 1 docs docs  35M Feb  9 13:25 GR.BFO..BHZ_2021.hdf5

[13]:

f = outdir.joinpath("GR.BFO..BHZ_2020-12.hdf5")
print(f.stat().st_size)

3059924

[14]:

f = outdir.joinpath("GR.BFO..BHZ_2020.hdf5")
print(f.stat().st_size)

36056084

Analyze processed data

With the analysis module, the processed data can be accessed and visualized.

[15]:

from data_quality_control import analysis #, util

[16]:

# Only for display in documentation!
from IPython.core.display import display, HTML

First, we initialize an Analyzer by setting the path to the HDF5-data (outdir), a station code and the fileunit, i.e. which name format the HDF5-files have, that we want to analyze.

The initial object does not have any data yet.

[17]:

reload(analysis)
#reload(util)
lyza = analysis.Analyzer(outdir, nslc_code,
                            fileunit="year")

24-02-09 13:25:48 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None

[18]:

print(lyza)

Analyzer for station GR.BFO..BHZ
Datadir: output/usage_demo
HDF5-file covers 1 year
Filename pattern: output/usage_demo/GR.BFO..BHZ_{year:04d}.hdf5
Loglevel: 10
No data attached.

We can inquire which files and time ranges are available for the given code, location and fileunit.

[19]:

files = lyza.get_available_datafiles()
print(files)

24-02-09 13:25:48 - data_quality_control.analysis.Analyzer - INFO - Looking for pattern output/usage_demo/GR.BFO..BHZ_[0-9][0-9][0-9][0-9].hdf5

['output/usage_demo/GR.BFO..BHZ_2021.hdf5', 'output/usage_demo/GR.BFO..BHZ_2020.hdf5']

[20]:

print(lyza.get_available_timerange())

24-02-09 13:25:48 - data_quality_control.analysis.Analyzer - INFO - Looking for pattern output/usage_demo/GR.BFO..BHZ_[0-9][0-9][0-9][0-9].hdf5
24-02-09 13:25:48 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:48 - data_quality_control.base.BaseProcessedData - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:49 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2020-12-25T01:00:00.000000Z - 2021-01-01T00:00:00.000000Z
24-02-09 13:25:49 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:49 - data_quality_control.base.BaseProcessedData - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:49 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2021-01-01T00:00:00.000000Z - 2021-01-09T23:00:00.000000Z

(UTCDateTime(2020, 12, 25, 1, 0), UTCDateTime(2021, 1, 9, 23, 0))

View data for time range

In order to view the data, the amplitudes and spectra are treated differently. Amplitudes are loaded as they are in the HDF5-file. Thus, we obtain an array of shape N_proclen x N_winlen. The sample data covers 16 days and we used proclen_seconds = 86400, i.e. 1 day, so the first dimension is 16. The winlen_seconds = 3600, thus 24 windows per day which gives the second dimension of the amplitude array.

For the spectra, there are two options: 1. load all spectra within a specific time range 2. load spectra for selected times given as list

The spectra are stored in the HDF5-files as 3D arrays. The first two dimensions correspond to those of the amplitude array; the third dimension is the frequency axis. In contrast, the Analyzer flattens the first to dimensions, i.e. the resulting array is basically a spectrogram, thus a sequence of spectra over time.

This allows to select spectra only for specific times. For example, one may want to select only those hours where the wind speed is in a specific range.

[21]:

startdate = UTC("2020-12-25")
enddate = UTC("2021-01-15")

[22]:

lyza.get_data(startdate, enddate)

24-02-09 13:25:49 - data_quality_control.analysis.Analyzer - INFO - Renewing data
24-02-09 13:25:49 - data_quality_control.analysis.Analyzer - INFO - Extending from file output/usage_demo/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:49 - data_quality_control.analysis.Analyzer - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:49 - data_quality_control.analysis.Analyzer - INFO - Extending from file output/usage_demo/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:49 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:49 - data_quality_control.base.BaseProcessedData - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:49 - data_quality_control.analysis.Analyzer - INFO - Start/end time set to: 2020-12-25T01:00:00.000000Z - 2021-01-09T23:00:00.000000Z
24-02-09 13:25:49 - data_quality_control.analysis.Analyzer - INFO - Adjusting starttime to available: 2020-12-25T01:00:00.000000Z
24-02-09 13:25:49 - data_quality_control.analysis.Analyzer - INFO - Adjusting endtime to available: 2021-01-09T23:00:00.000000Z

[23]:

print(lyza)
print("lyza is of type", type(lyza))

Analyzer for station GR.BFO..BHZ
Datadir: output/usage_demo
HDF5-file covers 1 year
Filename pattern: output/usage_demo/GR.BFO..BHZ_{year:04d}.hdf5
Loglevel: 10
I have data for 2020-12-25T01:00:00.000000Z - 2021-01-09T23:00:00.000000Z
Amplitude shape = (382,)
PSD shape = (382, 1025)
Seconds per window = 3600
Amplitude for 4 - 14 Hz
lyza is of type <class 'data_quality_control.analysis.Analyzer'>

Spectrogram

[24]:

fig = lyza.plot_spectrogram()
_images/01_usage_demo_36_0.png

Amplitudes

Amplitude values are visualized in a matrix covering date vs time of day (at least if you use appropriate processing and window length).

[25]:

lyza.plot_amplitudes();
_images/01_usage_demo_38_0.png

3D-Plots

Interactive 3D plots are created using plotly. The figures are HTML-code, heavily loaded with Javascript, which can be stored and viewed in a browser.

Careful!!! For large data sets, the files can become extremly large and your browser might not be able to handle it. So use with care.

[26]:

fig_amp, fig_psd = lyza.plot3d()

24-02-09 13:25:50 - data_quality_control.analysis.Analyzer - WARNING - Check size of HTML-figure! Your browser might crash!

[27]:

display(HTML(fig_psd.to_html(include_mathjax="cdn")))

[28]:

display(HTML(fig_amp.to_html(include_mathjax="cdn")))

In a notebook or script you could simply run:

fig.show()

View data for selected times

For some use cases one might want to get only the power spectral densities for specific times. For example, one could filter a time series of wind speed data for times with a certain speed.

The Analyzer extracts psds for specific times only if it receives a list of UTCDateTimes.

Note that for time lists the time axis in the spectral plots is only approximate!

Create random time list

For demonstration, we create a list of 100 random times within the time range of the data.

[29]:

starttime = UTC("2020-12-25")
endtime = UTC("2021-01-10")
times = np.arange(str(starttime.date), str(endtime.date),
             dtype="datetime64[h]")

[30]:

tlist = np.sort(np.random.choice(times, 100, replace=False))

tlist = [UTC(str(t)) for t in tlist]

Read data for times in list

Since we used 100 datetimes, we get a PSD-Matrix which has 100 entries along the time axis. The amplitude matrix remains unaffected.

[31]:

lyza.get_data(tlist)

print(lyza.startdate, lyza.enddate)
print(lyza.amplitudes.shape)
print("PSD shape:", lyza.psds.shape)
print("len(tlist):", len(tlist))

24-02-09 13:25:51 - data_quality_control.analysis.Analyzer - INFO - Renewing data
24-02-09 13:25:51 - data_quality_control.analysis.Analyzer - INFO - Extending from file output/usage_demo/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:51 - data_quality_control.analysis.Analyzer - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:51 - data_quality_control.analysis.Analyzer - INFO - Extending from file output/usage_demo/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:51 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:51 - data_quality_control.base.BaseProcessedData - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:51 - data_quality_control.analysis.Analyzer - INFO - Start/end time set to: 2020-12-25T01:00:00.000000Z - 2021-01-09T23:00:00.000000Z

2020-12-25T01:00:00.000000Z 2021-01-09T23:00:00.000000Z
(382,)
PSD shape: (99, 1025)
len(tlist): 100

[32]:

fig = lyza.plot_spectrogram()
_images/01_usage_demo_49_0.png

3D-Plots

[33]:

fig_amp, fig_psd = lyza.plot3d()

24-02-09 13:25:53 - data_quality_control.analysis.Analyzer - WARNING - Check size of HTML-figure! Your browser might crash!

[34]:

display(HTML(fig_psd.to_html(include_mathjax="cdn")))

[35]:

display(HTML(fig_amp.to_html(include_mathjax="cdn")))

Smooting & Data reduction

In some cases it might be desirable or even recommended to smooth and/or reduce the amount of data. For example, you computed PSDs and amplitude levels over 1h from the seismic data. If you tried to plot several years or even decades of data however, this would be still an enormeous amount of data which has to be plotted. (From experience, in a Jupyter notebook, even the matplotlib-figure (2D-plots) will freeze the browser; the 3D-figures can reach Gigabytes in size, so you probably won’t be able to open that either.)

Thus, at such a time range it is neither advisable not necessary to have such a time resolution. In that case, you can further reduce the already computed data using the SmoothOperator. It is build on top of the Analyzer which we used before, but applies a median filter operation on the data arrays. The filter operator is defined by 2 parameters. The kernel_size determines over how many samples the median is computed. The kernel_shift determines by how many samples the filter operator is shifted to compute the next median. Thus if kernel_shift=1 the data is just smoothed at the same resolution; if kernel_shift>1 the data is downsampled.

The result of this operation is a similar output file as from the original processing and is stored in the same way. You need to make sure to give a different directory, otherwise you existing data may be overwritten.

We create a new output directory for the smoothed data.

[36]:

smoothdir = outdir.joinpath("smoothed")
smoothdir.mkdir(exist_ok=True)

Now we initiate the operator. Note that the outdir from before becomes the data directory here. We set a kernel size of 6 samples, thus effectively 6h because the original data was processed over 3600s = 1h. The kernel is shift by 3 samples, thus in the end, we get a value per every 3 hours.

[37]:

reload(base)
reload(analysis)
polly = analysis.SmoothOperator(outdir, nslc_code,
                                kernel_size=6, kernel_shift=3)

24-02-09 13:25:54 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Set kernel_size=6, kernel_shift=3

Now we start the actual filtering.

[38]:

%%time
polly.smooth(smoothdir, force_new_file=True)

24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO -

Starting smooth()
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Looking up available timerange because start or endtime is None.

24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Looking for pattern output/usage_demo/GR.BFO..BHZ_[0-9][0-9][0-9][0-9].hdf5
24-02-09 13:25:54 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:54 - data_quality_control.base.BaseProcessedData - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:54 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2020-12-25T01:00:00.000000Z - 2021-01-01T00:00:00.000000Z
24-02-09 13:25:54 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:54 - data_quality_control.base.BaseProcessedData - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:54 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2021-01-01T00:00:00.000000Z - 2021-01-09T23:00:00.000000Z
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Renewing data
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Extending from file output/usage_demo/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Expecting window size is 3600s
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Set kernel_size=6, kernel_shift=3
24-02-09 13:25:54 - data_quality_control.base.ProcessedDataFileManager - INFO - Processed data stored per year in output/usage_demo/smoothed as {outdir}/{network}.{station}.{location}.{channel}_{year:04d}.hdf5
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO -

Iterating files over time range:

24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Interpolating 2020-12-25T00:00:00.000000Z - 2021-01-01T03:00:00.000000Z
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Renewing data
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Extending from file output/usage_demo/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:54 - data_quality_control.analysis.SmoothOperator - INFO - Extending from file output/usage_demo/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:54 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:54 - data_quality_control.base.BaseProcessedData - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:55 - data_quality_control.analysis.SmoothOperator - INFO - Start/end time set to: 2020-12-25T01:00:00.000000Z - 2021-01-09T23:00:00.000000Z
24-02-09 13:25:55 - data_quality_control.analysis.SmoothOperator - INFO - Adjusting starttime to available: 2020-12-25T01:00:00.000000Z
24-02-09 13:25:55 - data_quality_control.analysis.SmoothOperator - INFO - Start/end time set to: 2020-12-25T00:00:00.000000Z - 2021-01-01T03:00:00.000000Z
24-02-09 13:25:55 - data_quality_control.analysis.SmoothOperator - INFO - Setting time for output:
24-02-09 13:25:55 - data_quality_control.analysis.SmoothOperator - INFO - Start/end time set to: 2020-12-25T00:00:00.000000Z - 2021-01-01T00:00:00.000000Z
24-02-09 13:25:55 - data_quality_control.base.ProcessedDataFileManager - INFO - Creating output file output/usage_demo/smoothed/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:55 - data_quality_control.base.ProcessedDataFileManager - INFO - Starttime=2020-12-25T00:00:00.000000Z, endtime=2021-01-01T00:00:00.000000Z, n_windows=56
24-02-09 13:25:55 - data_quality_control.analysis.SmoothOperator - INFO - Interpolating 2021-01-01T00:00:00.000000Z - 2021-01-10T03:00:00.000000Z
24-02-09 13:25:55 - data_quality_control.analysis.SmoothOperator - INFO - Renewing data
24-02-09 13:25:55 - data_quality_control.analysis.SmoothOperator - INFO - Extending from file output/usage_demo/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:55 - data_quality_control.analysis.SmoothOperator - INFO - Reading file output/usage_demo/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:55 - data_quality_control.analysis.SmoothOperator - INFO - Start/end time set to: 2021-01-01T00:00:00.000000Z - 2021-01-10T03:00:00.000000Z

<HDF5 file "GR.BFO..BHZ_2020.hdf5" (mode r+)>

24-02-09 13:25:56 - data_quality_control.analysis.SmoothOperator - INFO - Setting time for output:
24-02-09 13:25:56 - data_quality_control.analysis.SmoothOperator - INFO - Start/end time set to: 2021-01-01T00:00:00.000000Z - 2021-01-10T00:00:00.000000Z
24-02-09 13:25:56 - data_quality_control.base.ProcessedDataFileManager - INFO - Creating output file output/usage_demo/smoothed/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:56 - data_quality_control.base.ProcessedDataFileManager - INFO - Starttime=2021-01-01T00:00:00.000000Z, endtime=2021-01-10T00:00:00.000000Z, n_windows=72
24-02-09 13:25:56 - data_quality_control.analysis.SmoothOperator - INFO - Interpolation finished.

<HDF5 file "GR.BFO..BHZ_2021.hdf5" (mode r+)>
CPU times: user 1.42 s, sys: 155 ms, total: 1.58 s
Wall time: 1.58 s

View smoothed data

[39]:

%%time
lyza = analysis.Analyzer(smoothdir, nslc_code, fileunit="year")

lyza.get_data(*lyza.get_available_timerange())

24-02-09 13:25:56 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:56 - data_quality_control.analysis.Analyzer - INFO - Looking for pattern output/usage_demo/smoothed/GR.BFO..BHZ_[0-9][0-9][0-9][0-9].hdf5
24-02-09 13:25:56 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:56 - data_quality_control.base.BaseProcessedData - INFO - Reading file output/usage_demo/smoothed/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:56 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2020-12-25T00:00:00.000000Z - 2021-01-01T00:00:00.000000Z
24-02-09 13:25:56 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:56 - data_quality_control.base.BaseProcessedData - INFO - Reading file output/usage_demo/smoothed/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:56 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: 2021-01-01T00:00:00.000000Z - 2021-01-10T00:00:00.000000Z
24-02-09 13:25:56 - data_quality_control.analysis.Analyzer - INFO - Renewing data
24-02-09 13:25:56 - data_quality_control.analysis.Analyzer - INFO - Extending from file output/usage_demo/smoothed/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:56 - data_quality_control.analysis.Analyzer - INFO - Reading file output/usage_demo/smoothed/GR.BFO..BHZ_2020.hdf5
24-02-09 13:25:56 - data_quality_control.analysis.Analyzer - INFO - Extending from file output/usage_demo/smoothed/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:56 - data_quality_control.base.BaseProcessedData - INFO - Start/end time set to: None - None
24-02-09 13:25:56 - data_quality_control.base.BaseProcessedData - INFO - Reading file output/usage_demo/smoothed/GR.BFO..BHZ_2021.hdf5
24-02-09 13:25:56 - data_quality_control.analysis.Analyzer - INFO - Start/end time set to: 2020-12-25T00:00:00.000000Z - 2021-01-10T00:00:00.000000Z

CPU times: user 22.9 ms, sys: 4.13 ms, total: 27 ms
Wall time: 26.1 ms

[40]:

print(lyza)

Analyzer for station GR.BFO..BHZ
Datadir: output/usage_demo/smoothed
HDF5-file covers 1 year
Filename pattern: output/usage_demo/smoothed/GR.BFO..BHZ_{year:04d}.hdf5
Loglevel: 10
I have data for 2020-12-25T00:00:00.000000Z - 2021-01-10T00:00:00.000000Z
Amplitude shape = (128,)
PSD shape = (128, 1025)
Seconds per window = 10800
Amplitude for 4 - 14 Hz

The spectrogram from the new data looks coarser then the first one.

[41]:

lyza.plot_spectrogram();
_images/01_usage_demo_65_0.png 
[42]:

lyza.plot_amplitudes();
_images/01_usage_demo_66_0.png 
[ ]: