Datasets:

---

Duplicated from  WenhaoWang/ScaleDF

ochanji

/

ScaleDF

Like 0

Error code:   DatasetGenerationError
Exception:    TypeError
Message:      int() argument must be a string, a bytes-like object or a real number, not 'NoneType'
Traceback:    Traceback (most recent call last):
                File "/usr/local/lib/python3.14/site-packages/datasets/builder.py", line 1531, in _prepare_split_single
                  for key, record in generator:
                                     ^^^^^^^^^
                File "/src/services/worker/src/worker/job_runners/config/parquet_and_info.py", line 613, in wrapped
                  for item in generator(*args, **kwargs):
                              ~~~~~~~~~^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 127, in _generate_examples
                  for example_idx, example in enumerate(self._get_pipeline_from_tar(tar_path, tar_iterator)):
                                              ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 32, in _get_pipeline_from_tar
                  for filename, f in tar_iterator:
                                     ^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/utils/track.py", line 49, in __iter__
                  for x in self.generator(*self.args):
                           ~~~~~~~~~~~~~~^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/utils/file_utils.py", line 1400, in _iter_from_urlpath
                  with xopen(urlpath, "rb", download_config=download_config, block_size=0) as f:
                       ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/utils/file_utils.py", line 977, in xopen
                  file_obj = fs.open(paths[0], mode)
                File "<string>", line 3, in open
                File "/usr/local/lib/python3.14/unittest/mock.py", line 1176, in __call__
                  return self._mock_call(*args, **kwargs)
                         ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/unittest/mock.py", line 1180, in _mock_call
                  return self._execute_mock_call(*args, **kwargs)
                         ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/unittest/mock.py", line 1247, in _execute_mock_call
                  result = effect(*args, **kwargs)
                File "/src/services/worker/src/worker/job_runners/config/parquet_and_info.py", line 786, in wrapped
                  tracker.files[urlpath] = {"read": 0, "size": int(f.size)}
                                                               ~~~^^^^^^^^
              TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'
              
              The above exception was the direct cause of the following exception:
              
              Traceback (most recent call last):
                File "/src/services/worker/src/worker/job_runners/config/parquet_and_info.py", line 1369, in compute_config_parquet_and_info_response
                  parquet_operations, partial, estimated_dataset_info = stream_convert_to_parquet(
                                                                        ~~~~~~~~~~~~~~~~~~~~~~~~~^
                      builder, max_dataset_size_bytes=max_dataset_size_bytes
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                  )
                  ^
                File "/src/services/worker/src/worker/job_runners/config/parquet_and_info.py", line 948, in stream_convert_to_parquet
                  builder._prepare_split(split_generator=splits_generators[split], file_format="parquet")
                  ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/builder.py", line 1393, in _prepare_split
                  for job_id, done, content in self._prepare_split_single(
                                               ~~~~~~~~~~~~~~~~~~~~~~~~~~^
                      gen_kwargs=gen_kwargs, job_id=job_id, **_prepare_split_args
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                  ):
                  ^
                File "/usr/local/lib/python3.14/site-packages/datasets/builder.py", line 1571, in _prepare_split_single
                  raise DatasetGenerationError("An error occurred while generating the dataset") from e
              datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset