Twenty’s Plenty: DataShare v2.1 Upload Upgrade

We have upgraded DataShare (to v2.1) to enable HTML5 resumable upload. This means depositors can now use the user-friendly web deposit interface to upload numerous files at once via drag’n’drop. And to upload files up to 15 GB in size, regardless of network ‘blips’.

In fact we have reason to believe it may be possible to upload a 20 GB file this way: in testing, I gave it 2 hours till the progress bar said 100%, and even though the browser then produced an error message instead of the green tick I was hoping for, I found when I retrieved the submission from the Submissions page that I was able to resume, and the file had been added.

*** So our new advice to depositors is: our current Item size limit and file size limit is 20 GB. Files larger than 15 GB may not upload through your browser. If you have files over 15 GB or data totalling over 20 GB which you’d like to share online, please contact the Data Library team to discuss your options. ***

See screenshots below. Once the files have been selected and the upload commenced, the ‘Status’ column shows the percentage uploaded. A 10 GB file may take in the region of 1 hour to upload in this way. 15 GB files have been uploaded with Chrome, Firefox and Internet Explorer using this interface.

Until now, any file over 1 GB had caused browsers difficulties, meaning many prospective depositors were not able to use the web deposit interface, and instead had to email the curation team, arrange to transfer us their files via DropBox, USB or through the Windows network, and then the curator had to transfer these same files to our server, collate the metadata into an XML file, log into the Linux system and run a batch import script. Often with many hiccups concerning permissions, virus checkers and memory along the way. All very time-consuming.

Soon we will begin working on a download upgrade, to integrate a means for users to download much bigger files from DataShare outside of the limitations of HTTP (perhaps using FTP). The aim is to allow some of the datasets we have in the university which are in the region of 100 GB to be shared online in a way that makes it reasonably quick and easy for users to download them. We have depositors queueing up to use this feature. Watch this space.

Further technical detail about both the HTML5 upload feature and plans for an optimised large download release are available on the slides for the presentation I made at Open Repositories 2016 in Dublin this week: http://www.slideshare.net/paulineward/growing-open-data-making-the-sharing-of-xxlsized-research-data-files-online-a-reality-using-edinburgh-datashare .

NewUploadPage

A simple interface invites the depositor to select files to upload.

 

 

Upload15GB

A 15 GB file uploaded via Firefox on Windows and included in a submitted Item.

 

 

A 20 GB file uploaded and included in an incomplete submission.

A 20 GB file uploaded and included in an incomplete submission.

Pauline Ward, Data Library Assistant, University of Edinburgh

Share

Highlights from the RDM Programme Progress Report: August – October 2015

The RDM Roadmap 2.0 has been completed, approved, and published online and work has started on achieving the deliverables. A copy of the Roadmap is publicly available on the RDM webpages and can be downloaded from http://www.ed.ac.uk/files/atoms/files//uoe-rdm-roadmap_-_v2_0.pdf.

The RDM Services brochure has now been published in both paper and electronic form and is proving very popular with researchers. The electronic version can be downloaded from http://www.ed.ac.uk/files/atoms/files/rdm_service_a5_booklet_0.pdf

Work on DataVault is progressing well and an interim DataVault service is now nearly complete. The Software Sustainability Institute has worked with the DataVault team to road test the interim solution, as a result some optimisations to the process were identified and are being coded up. DataVault user events have been held in both Manchester and Edinburgh, both events were well attended and the general impression of the current DataVault functionality was positive. Further, round three, funding is being sought from Jisc in December to continue this joint development effort.

Jisc has provided funding for up to nine PhD students to be employed one day per week for four months within their school. Their role will be to help researchers within their school record their research data as Datasets in the PURE system, and to direct any RDM or DMP queries to the RDM team for further support. The Dataset records in PURE will provide the Edinburgh University contribution to the national Research Data Discovery Service, this will increase the discoverability of Edinburgh data and ensure that more researchers are meeting the requirements of their research funders to make their data discoverable and reusable. Applications for the first set of three PhD student interns have been received and are currently being shortlisted, the successful applicants should be able to begin work before the end of 2015.

In October some minor questions were received about the DataShare application for Data Seal of Approval (DSA), these were responded to and DataShare has now been approved for the DSA. This is a major achievement for the entire DataShare team who have worked hard to make DataShare a Trusted Digital Repository.

Over the three month period a total of 173 staff and PGR’s have attended a RDM course or workshop, an additional 20-25 staff have attended research committee meetings or small group presentations where RDM has been on the agenda. Both regular and on demand RDM sessions (courses, workshops, & presentations) will continue to be offered and we are currently in the process of scheduling 30 courses, workshops for January to June 2016 as well as a number of presentations.

The “Data Management and Sharing” Coursera MOOC is well under way with a December launch anticipated. Sarah Jones, DCC, is our video instructor, using scripts adapted from MANTRA.

National and International Engagement Activities

10th August meeting in London with other Alan Turing Institute members to discuss RDM requirements to be provided by member institutions.

17th of August a one day RDM event was organised for Danish visitors from the University of Copenhagen to present UoE RDM services, outreach activities and ELNs.

31st August Dealing with Data conference.

7th/8th September meeting with Gottingen University to talk about digital scholarship, including RDM.

7th October DataVault engagement event at Manchester University.

29 October, Educause conference, Indianapolis. Robin Rice was on a panel with Jan Cheetham & Brianna Marshall, University of Wisconsin and Rory Macneil, RSpace: “Drivers and responses toward research data management maturity: transatlantic perspectives.

Kerry Miller

RDM Service Co-Ordinator

Share

Jisc Data Vault update

Posted on behalf of Claire Knowles

Research data are being generated at an ever-increasing rate. This brings challenges in how to store, analyse, and care for the data. Part of this problem is the long term stewardship of researchers’ private data and associated files that need a safe and secure home for the medium to long term.

PrintThe Data Vault project, funded by the Jisc #DataSpring programme seeks to define and develop a Data Vault software platform that will allow data creators to describe and store their data safely in one of the growing number of options for archival storage. This may include cloud solutions, shared storage systems, or local infrastructure.

Future users of the Data Vault are invited to Edinburgh on 5th November, to help shape the development work through discussions on: use cases, example data, retention policies, and metadata with the project team.

Book your place at: https://www.eventbrite.co.uk/e/data-vault-community-event-edinburgh-tickets-18900011443

The aims of the second phase of the project are to deliver a first complete version of the platform by the end of November, including:

  • Authentication and authorisation
  • Integration with more storage options
  • Management / monitoring interface
  • Example interface to CRIS (PURE)
  • Development of retention and review policy
  • Scalability testing

Working towards these goals the project team have had monthly face-to-face meetings, with regular Skype calls in between. The development work is progressing steadily, as you can see via the Github repository: https://github.com/DataVault, where there have now been over 300 commits. Progress is also tracked on the open Project Plan where anyone can add comments.

So remember, remember the 5th November and book your ticket.

Claire Knowles, Library & University Collections, on behalf of the JISC Data Vault Project Team

Share

Edinburgh DataShare – new features for users and depositors

I was asked recently on Twitter if our data library was still happily using DSpace for data – the topic of a 2009 presentation I gave at a DSpace User Group meeting. In responding (answer: yes!) I recalled that I’d intended to blog about some of the rich new features we’ve either adopted from the open source community or developed ourselves to deliver our data users and depositors a better service and fulfill deliverables in the University’s Research Data Management Roadmap.

Edinburgh DataShare was built as an output of the DISC-UK DataShare project, which explored pathways for academics to share their research data over the Internet at the Universities of Edinburgh, Oxford and Southampton (2007-2009). The repository is based on DSpace software, the most popular open source repository system in use, globally.  Managed by the Data Library team within Information Services, it is now a key component in the UoE’s Research Data Programme, endorsed by its academic-led steering group.

An open access, institutional data repository, Edinburgh DataShare currently holds 246 datasets across collections in 17 out of 22 communities (schools) of the University and is listed in the Re3data Registry of Research Data Repositories and indexed by Thomson-Reuters’ Data Citation Index.

Last autumn, the university joined DataCite, an international standards body that assigns persistent identifiers in the form of Digital Object Identifiers (DOIs) to datasets. DOIs are now assigned to every item in the repository, and are included in the citation that appears on each landing page. This helps to ensure that even after the DataShare system no longer exists, as long as the data have a home, the DOI will be able to direct the user to the new location. Just as importantly, it helps data creators gain credit for their published data through proper data citation in textual publications, including their own journal articles that explain the results of their data analyses.

CaptureThe autumn release also streamlined our batch ingest process to assist depositors with large and voluminous data files by getting around the web upload front-end. Currently we are able to accept files up to 10 GB in size but we are being challenged to allow ever greater file sizes.

Making the most of metadata

Discover panel screenshot

Example from Geosciences community

Every landing page (home, community, collection) now has a ‘Discover’ panel giving top hits for each metadata field (such as subject classification, keyword, funder, data type, spatial coverage). The panel acts as a filter when drilling down to different levels,  allowing the most common values to be ‘discovered’ within each section.

The usage statistics at each level  are now publicly viewable as well, so depositors and others can see how often an item is viewed or downloaded. This is useful for many reasons. Users can see what is most useful in the repository; depositors can see if their datasets are being used; stakeholders can compare the success of different communities. By being completely open and transparent, this is a step towards ‘alt-metrics’ or alternative ways measuring scholarly or scientific impact. The repository is now also part of IRUS-UK, (Institutional Repository Usage Statistics UK), which uses the COUNTER standard to make repository usage statistics nationally comparable.

What’s coming?

Stay tuned for future improvements around a new look and feel, preview and display by data type, streaming support, bittorent downloading, and Linked Open Data.

Robin Rice
EDINA and Data Library

Share