Can't read from one file then save to another file

[VBaratham]

1) Bug

Loading an NWBFile from one file, then saving it to another causes ValueError: Can't change container_source once set to be thrown

Steps to Reproduce

from datetime import datetime                                                                                         
from pynwb import NWBFile, NWBHDF5IO, TimeSeries                                                                   
                                                                                                                      
if __name__ == '__main__':                                                                                            
    nwb = NWBFile(source='', session_description='', identifier='', session_start_time=datetime.now())                                                                                    
    ts1 = TimeSeries(name='timeseries1', source='source1', data=[4,5,6],                                              
                     unit='na1', rate=1.0, starting_time=0.0)                                                         
    nwb.add_acquisition(ts1)                                                                                          
    with NWBHDF5IO('test_append.nwb', mode='w') as io:                                                                
        io.write(nwb)                                                                                                 
                                                                                                                      
    f = NWBHDF5IO('test_append.nwb')                                                                                  
    nwb = f.read()                                                                                                    
    ts2 = TimeSeries(name='timeseries2', source='source2', data=[4,5,6],                                              
                     unit='na1', rate=1.0, starting_time=0.0)                                                         
    nwb.add_acquisition(ts2)                                                                                          
    with NWBHDF5IO('test_append2.nwb', mode='a') as io:                                                               
        io.write(nwb)  

Environment

Please describe your environment according to the following bullet points.

• Python Executable: Conda
• Python Version: Python 2.7.15 and 3.5.1
• Operating System: Linux
• Pynwb Version: 0.5.1.post0.dev112

Checklist

• Have you ensured the feature or change was not already reported ?
• Have you included a brief and descriptive title?
• Have you included a clear description of the problem you are trying to solve?
• Have you included a minimal code snippet that reproduces the issue you are encountering?

[bendichter]

@VBaratham

You can write an external link to another object by using the manager argument of NWBHDF5IO. This writes an external link to a TimeSeries object in acquisition:

from datetime import datetime
from pynwb import NWBFile, NWBHDF5IO, TimeSeries, get_manager

manager = get_manager()

nwb = NWBFile(session_description='aa', identifier='aa', session_start_time=datetime.now())
ts1 = TimeSeries(name='timeseries1', data=[4, 5, 6],
                 unit='na1', rate=1.0, starting_time=0.0)
nwb.add_acquisition(ts1)
with NWBHDF5IO('test_append.nwb', mode='w', manager=manager) as io:
    io.write(nwb)

with NWBHDF5IO('test_append.nwb', 'r', manager=manager) as f:
    nwb_r = f.read()
    nwb = NWBFile(session_description='aa', identifier='aa', session_start_time=datetime.now())
    nwb.add_acquisition(nwb_r.acquisition['timeseries1'])
    with NWBHDF5IO('test_append2.nwb', mode='w', manager=manager) as io:
        io.write(nwb)

What you describe- linking an entire NWBFile object- would at its simplest look like this:

from datetime import datetime
from pynwb import NWBFile, NWBHDF5IO, get_manager

manager = get_manager()

nwb = NWBFile(session_description='aa', identifier='aa', session_start_time=datetime.now())
with NWBHDF5IO('file1.nwb', mode='w', manager=manager) as io:
    io.write(nwb)
with NWBHDF5IO('file1.nwb', 'r', manager=manager) as f:
    nwb = f.read()
    with NWBHDF5IO('file2.nwb', mode='w', manager=manager) as io:
        io.write(nwb)

This would instruct pynwb to write the entire file as an external link to another file. You are correct that this leads to an error, but is this really what you are trying to do?

[VBaratham]

Hi @bendichter

I think I'm a bit lost. Here's what I want to do:

I have a file called test.nwb. I want to read that in, add some data or something, and then save it out to a file called test2.nwb so that I have two nwb files side by side, the first of which (test.nwb) is missing this added dataset. I don't want the second file to be a link to the first, I want it to have all the data replicated. Is this possible? Or do I have to use some other tool to duplicate the file before adding to one of them (would that work)?

[oruebel]

@VBaratham by default PyNWB will in this case create external links in test2.nwb to the containers that have previously been written to test.nwb. If you want all the data from test.nwb to be copied to test2.nwb then simply set link_data=False on write which will change the default behavior to copy the data rather than link it. I.e,

with NWBHDF5IO('file2.nwb', mode='w', manager=manager) as io:
        io.write(nwb, link_data=False)

[bendichter]

Thanks for the clarification, Oliver. We should add that to the tutorials

[oruebel]

@bendichter what do you think the best place for this is. Maybe in the modular storage tutorial:

https://pynwb.readthedocs.io/en/stable/tutorials/general/linking_data.html#sphx-glr-tutorials-general-linking-data-py

[bendichter]

Yeah or maybe in advanced IO

[t-b]

~/devel/pynwb/docs/gallery/domain (master) $ git diff .
diff --git a/docs/gallery/domain/icephys.py b/docs/gallery/domain/icephys.py
index 965270ed..82a07c3b 100644
--- a/docs/gallery/domain/icephys.py
+++ b/docs/gallery/domain/icephys.py
@@ -150,3 +150,11 @@ elec = ccss.electrode
 # And the device name via :py:meth:`~pynbwb.file.NWBFile.get_device`

 device = nwbfile.get_device('Heka ITC-1600')
+
+io.close()
+
+io = NWBHDF5IO('icephys_example.nwb', 'w')
+nwbfile = io.read()
+nwbfile.add_acquisition(vcs)
+io.write(nwbfile)
+io.close()

~/devel/pynwb/docs/gallery/domain (master) $ python icephys.py
Traceback (most recent call last):
  File "icephys.py", line 157, in <module>
    nwbfile = io.read()
  File "/home/firma/.local/lib/python2.7/site-packages/hdmf/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/home/firma/.local/lib/python2.7/site-packages/hdmf/backends/io.py", line 33, in read
    container = self.__manager.construct(f_builder)
  File "/home/firma/.local/lib/python2.7/site-packages/hdmf/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/home/firma/.local/lib/python2.7/site-packages/hdmf/build/map.py", line 195, in construct
    result = self.__type_map.construct(builder, self)
  File "/home/firma/.local/lib/python2.7/site-packages/hdmf/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/home/firma/.local/lib/python2.7/site-packages/hdmf/build/map.py", line 1561, in construct
    attr_map = self.get_map(builder)
  File "/home/firma/.local/lib/python2.7/site-packages/hdmf/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/home/firma/.local/lib/python2.7/site-packages/hdmf/build/map.py", line 1492, in get_map
    container_cls = self.get_cls(obj)
  File "/home/firma/.local/lib/python2.7/site-packages/hdmf/utils.py", line 381, in func_call
    return func(self, **parsed['args'])
  File "/home/firma/.local/lib/python2.7/site-packages/hdmf/build/map.py", line 1424, in get_cls
    raise ValueError("No data_type found for builder %s" % builder.path)
ValueError: No data_type found for builder root

~/devel/pynwb/docs/gallery/domain (master) $ git rev-parse HEAD
8f472d6c5b74a72e57073fca4c8100a60dfc16a1
~/devel/pynwb/docs/gallery/domain (master) $

This has nothing to do with external links.

[rly]

@t-b in your example, you are opening the file in write mode but then reading from it, which is not allowed. We should check if that happens and throw a user-friendly error if so.

If you change 'w' to 'r+', then you get the correct error:

Traceback (most recent call last):
  File "icephys.py", line 159, in <module>
    nwbfile.add_acquisition(vcs)
  File "c:\users\ryan\documents\nwb\hdmf\src\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "c:\users\ryan\documents\nwb\pynwb\src\pynwb\file.py", line 617, in add_acquisition
    self._add_acquisition_internal(nwbdata)
  File "c:\users\ryan\documents\nwb\hdmf\src\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "c:\users\ryan\documents\nwb\pynwb\src\pynwb\core.py", line 743, in _func
    raise ValueError(msg)
ValueError: 'vcs' already exists in 'root'

[t-b]

@rly Thanks, granted this was sloopy.

Next try:

~/devel/pynwb (better-validation) $ git diff .
diff --git a/docs/gallery/domain/icephys.py b/docs/gallery/domain/icephys.py
index bfff2ed6..448f529a 100644
--- a/docs/gallery/domain/icephys.py
+++ b/docs/gallery/domain/icephys.py
@@ -150,3 +150,22 @@ elec = ccss.electrode
 # And the device name via :py:meth:`~pynbwb.file.NWBFile.get_device`

 device = nwbfile.get_device('Heka ITC-1600')
+
+io.close()
+
+ioread = NWBHDF5IO('icephys_example.nwb', 'r')
+
+vcs2 = VoltageClampSeries(
+    name='vcs2', data=[0.1, 0.2, 0.3, 0.4, 0.5],
+    unit='A', conversion=1e-12, resolution=np.nan, starting_time=123.6, rate=20e3,
+    electrode=elec, gain=0.02, capacitance_slow=100e-12, resistance_comp_correction=70.0,
+    capacitance_fast=np.nan, resistance_comp_bandwidth=np.nan, resistance_comp_prediction=np.nan,
+    whole_cell_capacitance_comp=np.nan, whole_cell_series_resistance_comp=np.nan)
+
+nwbfile = ioread.read()
+nwbfile.add_acquisition(vcs2)
+
+io = NWBHDF5IO('icephys_example2.nwb', 'w')
+io.write(nwbfile)
+io.close()
+ioread.close()

~/devel/pynwb (better-validation) $ python docs/gallery/domain/icephys.py
Traceback (most recent call last):
  File "docs/gallery/domain/icephys.py", line 169, in <module>
    io.write(nwbfile)
  File "/home/firma/devel/hdmf/src/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/firma/devel/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 196, in write
    call_docval_func(super(HDF5IO, self).write, kwargs)
  File "/home/firma/devel/hdmf/src/hdmf/utils.py", line 281, in call_docval_func
    return func(*fargs, **fkwargs)
  File "/home/firma/devel/hdmf/src/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/firma/devel/hdmf/src/hdmf/backends/io.py", line 39, in write
    f_builder = self.__manager.build(container, source=self.__source)
  File "/home/firma/devel/hdmf/src/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/firma/devel/hdmf/src/hdmf/build/map.py", line 158, in build
    raise ValueError("Can't change container_source once set")
ValueError: Can't change container_source once set
~/devel/pynwb (better-validation) $

[luiztauffer]

@oruebel @bendichter
The problem seems to persist, even with link_data=False. Here's a reproducible error:

from datetime import datetime
from dateutil.tz import tzlocal
from pynwb import NWBFile, NWBHDF5IO, get_manager, TimeSeries
import nwbext_ecog
import numpy as np
import os

manager = get_manager()

# Creates file 1
nwb_1 = NWBFile(session_description='session', identifier='1', session_start_time=datetime.now(tzlocal()))
# Add signal
X_data = np.zeros((10,1000))
timestamps = list(range(1000))
test_ts = TimeSeries(name='test_timeseries', data=X_data, unit='s', timestamps=timestamps)
nwb_1.add_acquisition(test_ts)
with NWBHDF5IO('file_1.nwb', mode='w') as io:
    io.write(nwb_1)


# Open file 1
with NWBHDF5IO('file_1.nwb', 'r', manager=manager) as io_1:
    nwb_1 = io_1.read()
    # Open file 2
    with NWBHDF5IO('file_2.nwb', mode='w', manager=manager) as io_2:
        nwb_2 = NWBFile(session_description='session', identifier='2', session_start_time=datetime.now(tzlocal()))
        for aux in nwb_1.acquisition:      # Acquisition
            nwb_2.add_acquisition(nwb_1.get_acquisition(aux))
        io_2.write(nwb_2, link_data=False)
        print(nwb_2.acquisition['test_timeseries'])

file_2.nwb was created and apparently has something in the acquisition field. However, the file sizes are different and, when I delete file_1.nwb, reading from file_2.nwb raises and error.

# Open file 2 after deleting file 1
with NWBHDF5IO('file_2.nwb', 'r', manager=manager) as io:
    nwb_2 = io.read()
    print(nwb_2.acquisition['test_timeseries'])

AttributeError: 'NoneType' object has no attribute 'name'