Write HDF5 Error for large matrix #702

Goon83 · 2020-03-26T18:07:54Z

Hi All,
Hope you are doing well under current virus epidemic situation. I understand that you may have important things to do now. I just posted an error information here in case you can get chance to look into it.

Recently, I tested the function StoreHDF::write and found out that it works on small matrix but has issue for large array on multiple processes (CPU units). Below is the code:

using dash::io::hdf5::hdf5_options;
using dash::io::hdf5::StoreHDF;

#define N1 201
#define N2 15000
int main(int argc, char *argv[])
{
	dash::init(&argc, &argv);

	dash::Matrix<double, 2> *h5matrix = new dash::Matrix<double, 2>(dash::SizeSpec<2>(N1, N2));

	auto myid = dash::myid();

	if (!myid)
	{
		for (int i = 0; i < N1; i++)
		{
			for (int j = 0; j < N2; j++)
				h5matrix->at(i, j) = i + j;
		}
	}
	StoreHDF::write(*h5matrix, "testf.h5", "testg/testd2D");
	StoreHDF::read(*h5matrix, "testf.h5", "testg/testd2D");

	if (myid == 1)
	{
		for (int i = 0; i < N1; i++)
		{
			for (int j = 0; j < N2; j++)
			{
				double t = h5matrix->at(i, j);
				if (t != (i + j))
				{
					std::cout << "Wrong result \n";
					exit(-1);
				}
			}
		}
	}
	dash::finalize();

	return 0;
}

I compiled the code and ran it with 2 process, and it reports below error. Note that, if you change N1 and N2 in the code to small number, e.g, 10 by 10. It works.

>> mpirun  -n 2 ./h5-test
HDF5-DIAG: Error detected in HDF5 (1.10.5) MPI-process 1:
  #000: H5Dio.c line 322 in H5Dwrite(): could not get a validated dataspace from file_space_id
    major: Invalid arguments to routine
    minor: Bad value
  #001: H5S.c line 254 in H5S_get_validated_dataspace(): selection + offset not within extent
    major: Dataspace
    minor: Out of range
^C[mpiexec@dbinMac] Sending Ctrl-C to processes as requested
[mpiexec@dbinMac] Press Ctrl-C again to force abort

The text was updated successfully, but these errors were encountered:

Goon83 · 2020-06-12T16:54:20Z

Hi DASH Community,
Just check whether someone can help to check this issue ?

Bests,
Bin

dhinf · 2020-06-17T10:39:33Z

I will look into it this week.

dhinf · 2020-06-18T10:41:42Z

The problem is not the total size of the NArray, it is the size of the extension. E.g. 21 x 20 also results in an error. For 21 a the first rank has 11 elements in the first dimension, but the second one only 20. This seems to be a problem. I tried it with the outputstream. In the end it should be the same result.

#define N1 200
#define N2 15000

#define FILENAME "example.hdf5"

int main(int argc, char *argv[])
{
  dash::init(&argc, &argv);
  dash::Matrix<double, 2> h5matrix(dash::SizeSpec<2>(N1, N2));
  auto myid = dash::myid();

  if (!myid) {
    for (int i = 0; i < N1; i++) {
	for (int j = 0; j < N2; j++)
	  h5matrix.at(i, j) = i + j;
    }
  }
  dash::io::hdf5::OutputStream os(FILENAME);
  os << dash::io::hdf5::dataset("group/data") << h5matrix;

  dash::barrier();
  if(dash::myid() == 0){
    std::string syscall = "h5dump ";
    auto status = system((syscall + FILENAME).c_str());
  }
  dash::finalize();

 return 0;
}

dhinf · 2020-06-19T11:10:49Z

It is a bug in the TilePattern. When you use the proxy dash::NArray instead of dash::Matrix it should work. dash::Matrix uses the per default the TilePattern while dash::Narray uses a BlockPattern instead. That's the only difference.

dhinf · 2020-06-19T11:11:20Z

little work around until we fixed the pattern

dhinf · 2020-06-19T15:37:23Z

I fixed it, but if you compile dash with enabled assertions you will get an error by using the TilePattern with underfilled blocks.
@devreal and @fuchsto: Why does a TilePattern can't have underfilled blocks? What was the reason to forbid it.

devreal · 2020-06-19T16:16:00Z

I believe that is a longstanding issue that has never been properly implemented. If someone has a patch I would love that...

dhinf · 2020-06-22T08:54:41Z

The solution would be the same as for the BlockedPattern. Only the last Block is underfilled. If that is fine i will open a pull request.

devreal · 2020-06-22T09:12:11Z

Absolutely, please give it a shot 👍

dhinf · 2020-06-26T15:17:05Z

fixed with pr #713

Goon83 · 2021-01-05T23:28:30Z

@dhinf @devreal

Thanks for working on this issue.
Tested the bug-dash-hdf5-pattern branch and it works.

Could you please review the merge and get the code into development branch?

Thanks.
Bin

Goon83 · 2021-04-26T05:14:51Z

@dhinf @devreal

I recently tested the code on a 1D data and found the StoreHDF::write and StoreHDF::read still can not work.
The test cod code and error information are presented in below.
Could you help to look into this?

Bests,
Bin
Test code:

#include "libdash.h"
#include

using dash::io::hdf5::hdf5_options;
using dash::io::hdf5::StoreHDF;

#define N1 201

int main(int argc, char *argv[])
{
dash::Matrix<double, 1> *h5matrix_1d = new dash::Matrix<double, 1>(dash::SizeSpec<1>(N1));
auto myid = dash::myid();

if (!myid)
{
    for (int i = 0; i < N1; i++)
    {
        h5matrix_1d->at(i) = i;
    }
}
StoreHDF::write(*h5matrix_1d, "testf-1d.h5", "testg/testd1D");
StoreHDF::read(*h5matrix_1d, "testf-1d.h5", "testg/testd1D");

if (myid == 1)
{
    for (int i = 0; i < N1; i++)
    {
        double t = h5matrix_1d->at(i);
        if (t != i)
        {
            std::cout << "Wrong result \n";
            exit(-1);
        }
    }
}

dash::finalize();

return 0;

}

==========
Error Info:

dbin@Bins-MBP dash % ~/work/soft/dash/build/install/bin/dash-mpiCC h5-1d.cpp -o h5-1d
In file included from h5-1d.cpp:1:
In file included from /Users/dbin/work/soft/dash/build/install//include/libdash.h:71:
In file included from /Users/dbin/work/soft/dash/build/install//include/dash/io/HDF5.h:4:
/Users/dbin/work/soft/dash/build/install//include/dash/io/hdf5/StorageDriver.h:566:26: error: no member named
'underfilled_blocksize' in 'dash::TilePattern<1, dash::ROW_MAJOR, long>'
} else if (pattern.underfilled_blocksize(dimensions.back()) == 0) {
~~~~~~~ ^
/Users/dbin/work/soft/dash/build/install//include/dash/io/hdf5/StorageDriver.h:600:12: note: in instantiation of function
template specialization 'dash::io::hdf5::StoreHDF::_get_hdf_slabs_with_underfilled<dash::TilePattern<1,
dash::ROW_MAJOR, long> >' requested here
return _get_hdf_slabs_with_underfilled(pattern);
^
/Users/dbin/work/soft/dash/build/install//include/dash/io/hdf5/internal/DriverImplZeroCopy.h:32:21: note: in instantiation
of function template specialization 'dash::io::hdf5::StoreHDF::_get_hdf_slabs<1, dash::ROW_MAJOR, long>' requested
here
auto hyperslabs = _get_hdf_slabs(container.pattern());
^
/Users/dbin/work/soft/dash/build/install//include/dash/io/hdf5/StorageDriver.h:762:5: note: in instantiation of function
template specialization 'dash::io::hdf5::StoreHDF::_process_dataset_impl_zero_copy<dash::Matrix<double, 1, long,
dash::TilePattern<1, dash::ROW_MAJOR, long>, dash::HostSpace> >' requested here
_process_dataset_impl_zero_copy(StoreHDF::Mode::WRITE, container, h5dset,
^
/Users/dbin/work/soft/dash/build/install//include/dash/io/hdf5/StorageDriver.h:234:5: note: in instantiation of function
template specialization 'dash::io::hdf5::StoreHDF::_write_dataset_impl<dash::Matrix<double, 1, long,
dash::TilePattern<1, dash::ROW_MAJOR, long>, dash::HostSpace> >' requested here
_write_dataset_impl(array, h5dset, internal_type);
^
h5-1d.cpp:21:15: note: in instantiation of function template specialization
'dash::io::hdf5::StoreHDF::write<dash::Matrix<double, 1, long, dash::TilePattern<1, dash::ROW_MAJOR, long>,
dash::HostSpace> >' requested here
StoreHDF::write(*h5matrix_1d, "testf-1d.h5", "testg/testd1D");
^
1 error generated.

dhinf · 2021-04-27T14:03:52Z

@Goon83

i'll look into it. I need to add the missing method inside the pattern. i'll try to do it this week.

best
Denis

dhinf · 2021-04-28T08:26:19Z

It is fixed and merged to development. Btw. in your example code dash::init(&argc,&argv) is missing.
Please check, if the fix works your environment.

dhinf assigned fuchsto Jun 19, 2020

dhinf added help wanted specification:dash-concepts labels Jun 19, 2020

dhinf mentioned this issue Apr 27, 2021

added underfilled blocks to 1d Tile Pattern #718

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write HDF5 Error for large matrix #702

Write HDF5 Error for large matrix #702

Goon83 commented Mar 26, 2020 •

edited

Loading

Goon83 commented Jun 12, 2020

dhinf commented Jun 17, 2020

dhinf commented Jun 18, 2020 •

edited

Loading

dhinf commented Jun 19, 2020 •

edited

Loading

dhinf commented Jun 19, 2020

dhinf commented Jun 19, 2020

devreal commented Jun 19, 2020

dhinf commented Jun 22, 2020

devreal commented Jun 22, 2020

dhinf commented Jun 26, 2020

Goon83 commented Jan 5, 2021 •

edited

Loading

Goon83 commented Apr 26, 2021

dhinf commented Apr 27, 2021 •

edited

Loading

dhinf commented Apr 28, 2021 •

edited

Loading

Write HDF5 Error for large matrix #702

Write HDF5 Error for large matrix #702

Comments

Goon83 commented Mar 26, 2020 • edited Loading

Goon83 commented Jun 12, 2020

dhinf commented Jun 17, 2020

dhinf commented Jun 18, 2020 • edited Loading

dhinf commented Jun 19, 2020 • edited Loading

dhinf commented Jun 19, 2020

dhinf commented Jun 19, 2020

devreal commented Jun 19, 2020

dhinf commented Jun 22, 2020

devreal commented Jun 22, 2020

dhinf commented Jun 26, 2020

Goon83 commented Jan 5, 2021 • edited Loading

Goon83 commented Apr 26, 2021

Bests, Bin Test code:

========== Error Info:

dhinf commented Apr 27, 2021 • edited Loading

dhinf commented Apr 28, 2021 • edited Loading

Goon83 commented Mar 26, 2020 •

edited

Loading

dhinf commented Jun 18, 2020 •

edited

Loading

dhinf commented Jun 19, 2020 •

edited

Loading

Goon83 commented Jan 5, 2021 •

edited

Loading

Bests,
Bin
Test code:

==========
Error Info:

dhinf commented Apr 27, 2021 •

edited

Loading

dhinf commented Apr 28, 2021 •

edited

Loading