athena.data.datasets.base

base dataset

Module Contents

Classes

BaseDatasetBuilder	base dataset builder
SpeechBaseDatasetBuilder	speech base dataset

Functions

data_loader(dataset_builder[, batch_size, num_threads])

data loader

athena.data.datasets.base.data_loader(dataset_builder, batch_size=16, num_threads=1)

data loader

class athena.data.datasets.base.BaseDatasetBuilder(config=None)

base dataset builder

abstract property sample_type

example types

abstract property sample_shape

examples shapes

abstract property sample_signature

examples signature

default_config

reload_config(config)

reload the config

abstract preprocess_data(file_path)

loading data

abstract __getitem__(index)

__len__()

as_dataset(batch_size=16, num_threads=1)

return tf.data.Dataset object

shard(num_shards, index)

creates a Dataset that includes only 1/num_shards of this dataset

batch_wise_shuffle(batch_size=64, epoch=-1, seed=917)

Batch-wise shuffling of the data entries.

Each data entry is in the format of (audio_file, file_size, transcript). If epoch_index is 0 and sortagrad is true, we don’t perform shuffling and return entries in sorted file_size order. Otherwise, do batch_wise shuffling.

Parameters

batch_size (int, optional) – an integer for the batch size. Defaults to 64.

compute_cmvn_if_necessary(is_necessary=True)

compute cmvn file

class athena.data.datasets.base.SpeechBaseDatasetBuilder(config=None)

Bases: BaseDatasetBuilder

speech base dataset

abstract property num_class

return the number of classes

default_config

compute_cmvn_if_necessary(is_necessary=True)

vitural interface