Cloud Computing and Bioinformatics

RDA: a global research data orchestra

Fri, 30 Mar 2018 00:00:00 +0300

March 21st 2018; that was the day of the spring equinox this year, but also the first day of the 11th RDA Plenary, marking the 5th birthday of the Research Data Alliance. That may sound as a coincidence (and in all honesty it probably was), but it was definitely a great way to kick off the major biannual event of all research data aficionados gathered in Berlin. Yours truly was one of the 661 participants from 41 countries of the event that spanned three days and included more than 80 sessions, working meetings, BoFs and, of course, lots of caffeine-driven discussions in the halls of the Berlin Congress Center.

One of the major themes that resonated in this Plenary, was the concept of the global community; the group of approximately 6,600 people from all around the world that work together to address the challenges of open research data sharing. Open Science principles, interdisciplinary initiatives and targeted skill training where just few of the topics identified as global issues that are being currently tackled by the joined RDA community. Under the theme “From Data to Knowledge”, speakers throughout the Plenary provided some insights on these topics, as well as some suggestions for the way ahead.

Dr. Georg Schütte, State Secretary at the Federal Ministry of Education and Research in Germany, succinctly stated the current situation: “Science and research are at the forefront of the data revolution”. Interdisciplinary science, a recurring and familiar cadence in research, is deeply complex and takes time. Understanding the distinct roles, strengths, and contributions within a global community, can facilitate a culture of multi-disciplinary research. On the same note, Prof. Li Jianhui, the Secretary-General of the Chinese National Committee for the Committee on Data for Science and Technology (CODATA-CHINA) and Director of the Big Data Department of the CAS Computer Network Information Center (CNIC), emphasized the investment and progress in data science and data integration in China, but at the same time highlighted the need for common data tools and standards. A targeted session on the Open Science Commons provided the platform for further delving into this topic; hearing about developments in Africa, Australia, Canada, Europe and US (Nick Weber) set the stage for discussions towards a Global Open Science Commons. The main message here was to establish a feedback loop, from global to local and back again; best practices and solutions need to be identified globally, applied and refined at the local level and then reinforced again through global coordination. This dynamic interaction can lead to the standardization of protocols that can address some of the existing technical (and cultural) challenges.

On the other hand, as a community we need to be also mindful of the existing initiatives in the different countries and how they can be interoperable. There are several such initiatives, and at different levels of relevance; the G7 communique on Open Science in 2017 attempted to set a global tone; the “All of Us” programme of NIH Data Science, as outlined by Patti Brennan, Director of the National Library of Medicine , aims to gather data from over 1 million participants in order to offer a broad repository of content that’s not focused on specific disease investigation. At the European level, Prof. Klaus Tochtermann, Director of ZBW, put it quite elegantly: “We need to ensure that EOSC doesn’t become synonymous to the “European Open Science Chaos” “ Using FAIR principles, the primary focus of the GO FAIR initiative, can be a good step in this direction.

Going beyond the technical aspects, the human factor is equally important - what are the incentive structures in place for scientists to promote open and FAIR data? The need and advantages are clear to all, but the actual application and uptake is still minimal and sporadic at best. An interesting tidbit of information was highlighed during one of the sessions at P11:

Percentage of time spent finding and organising data according to research data specialists: 79%

This is why discussions on FAIR data are important, and it clearly highlights the need of building data skills globally.

I will close this meandering article by recalling (and slightly paraphrasing) an awesome metaphor given by the RDA Secretary General Hilary Hanahoe during the opening talk with regards to the RDA community.

Our community is the driving force of RDA. In a way similar to an orchestra, every member of the community is an instrument; each brings their own skillset, experience and particular expertise, producing a unique melody. Combining all these voices and melodies together, the music of RDA is produced.

And it’s beautiful. 🎼

September 2016 – the final tally

Sat, 15 Oct 2016 00:00:00 +0300

Well, I will definitely remember September 2016 for a number of reasons, but one is definitely the amount of travel involved. The story will be much easier if I break this down to numbers:

: Morning view, between sea and sky

Nights away from home: 17
Total Distance traveled: 31,444 km (fyi, this 78% of the circumference of the earth)
- By airplane: 30,711 km in 15 flights
- By car: 448 km
- On foot: 193 km (curtesy of my tracker)
- By train: 92 km
Locations
- 5 Cities: The Hague, Denver, Warsaw, Krakow, Trikala
- 8 Airport Codes: SKG, ATH, AMS, MUC, DEN, WAW, VIE, KRK
- 3 Countries: The Netherlands, USA, Poland
- 2 Continents: Europe, America

All in all, a very active month (but I’d rather not do so many trips in that short of a time). :)

Teaching my first Software Carpentry Workshop

Thu, 06 Oct 2016 00:00:00 +0300

“My first Software Carpentry workshop”. Sounds a bit like a first-grader’s essay. But in all honesty, it was a brand new experience for me. Sure, I have been teaching University lectures and seminars for quite some time, but doing a live-coding session was an experience of its own.

: SWC Workshop @ Thessaloniki

All things considered, the workshop was a success (evident also in the group photo here); participants were engaged throughout the two-day workshop, there was continuous feedback and discussions on tough points in the lesson, so everyone left quite happy and (hopefully) more knowledgeable than before.

For me, two things have become even more clear:

Better organization of the workshop in terms of the participants’ level of experience. Sure, very few actually complained for the pace, but I have the impression that we could do a better job if we had focused only on beginner level (for example) or intermediate.
Two days may be convenient in terms of time allocated for any person (given the rather busy schedule of most people now), but it’s a very short time to cover R, Shell and Git. So maybe next time (because there will definitely be a next time) we will aim for three days.

Oh, and a final piece of advice: if you intend to have food or catering at a SWC workshop, ordering pizza may be the best solution…. 🙂

Training for NGS data analysis using Chipster

Thu, 22 Oct 2015 00:00:00 +0300

The story is rather simple. Yesterday, my lab together with the Institute of Applied Biosciences co-organized a training workshop for NGS data analysis. For anyone even remotely engaged in NGS data, the biggest problem in NGS data is usually the computational complexity. In simple words, analyzing tons of data takes a very very long time. Which means that essentially the analysis in performed by people that are familiar with the tools (and their command-line interfaces) that can be used in high end computational systems.

However, this workshop went slightly off the treaded path by (mostly) skipping the command line interface and going directly to the graphical interface of Chipster, developed, maintained and kindly provided by CSC. This “deviation” allowed the participants, who had mainly wet-lab research background, to easily follow the established workflows and pipelines used in NGS data analysis. Moreover, instead of using local computational resources, we launched several Chipster servers through the EGI Federated Cloud. So in one training session, the participants were exposed both to the computational capabilities and infrastructure of EGI, as well as the pipelines used in NGS data analysis. All in all, a very dense 8-hour workshop!

The level of the participants’ experience was also quite diverse, ranging from undergraduate students to faculty members and staff scientists. Despite that though, the workshop was very engaging to all members, a fact clearly seen in the happy faces all around, even when the workshop extended a full hour beyond the expected wrap-up time!

So, the take home message; there is clearly a need (some might consider it a desperate one) for training events in bioinformatics, and especially in Big Data studies such as NGS data analysis. However, such events should not necessarily focus on the tech-savvy user. Or at least, actively encourage the non technical-expert researchers to attend by providing (a) user friendly interfaces, (b) hands-on exercises that feel close to the actual work of the participants, and (c) the time necessary for everyone to keep their own pace.

Finally, I would be remiss if I didn’t thank enough the two people that really supported this workshop: Diego Scardaci from EGI.eu and Kimmo Mattila from CSC, whom I constantly pestered with questions and issues in the past few weeks, and they always had the time and patience to lend me their experience.

Hopefully, there will be follow-up and more specialized workshops. However, if you are interested, the next one will take place at the EGI Community Forum in Bari. So, hope to see you there!

Integrating datasets for bioinformatics

Wed, 18 Feb 2015 00:00:00 +0200

Well, it seems that I have yet another story regarding EGI. Actually make that two; one is a new article on the EGI Inspire Newsletter in collaboration with Rafael Jimenez regarding a joint project between EGI and ELIXIR. The second is this joint project.

A collaboration between ELIXIR and EGI is by itself great news. Personally it means that there will be greater opportunities to find (and probably develop) bioinformatics tools that will also utilize and work with the computational infrastructure of EGI. And with little to no expertise required from the end user; it’s no secret that the average wet-lab researcher is a bit hesitant went it comes down to the “little black window” a.k.a. terminal. :)

The joint project I mentioned earlier is one that I am proud of being the coordinator of. It is an EGI Virtual Team project on Integrating Life Science Reference Datasets. Yes, I know it’s a mouthful but it’s really quite simple: instead of having to constantly copy reference datasets (i.e. NR/NT, UniProt, BowTie index files etc) to several computational nodes, leave it to the infrastructure to do it for you.

As a project it just started, but hopefully we’ll have some interesting results in the next 9 months. I’ll keep you posted!

Future opportunities and trends for e-infrastructures and life sciences

Wed, 22 Jan 2014 00:00:00 +0200

Working with friends, beyond being a pleasure, usually bears fruit. Case in study, the article published today on the EGI Inspire Newsletter with the help of close friend and colleague Afonso Duarte.

Beyond being a nice study on the trend of Life Sciences working with e-infrastructures (and Grid/Cloud computing specifically), this article is also an announcement of the Workshop we are organizing in the upcoming EGI Conference in Helsinki. Hope to see you there too! ;)

Perl DANCER framework

Wed, 31 Jul 2013 00:00:00 +0300

After quite some time, I had to go back to writing web-services. I can’t say I was really looking forward to it, but don’t get me wrong; I love writing code and I feel quite confident at it. On the other hand, my memories in writing web-services aren’t the most comforting ones. Admittedly, last time was in 2008 and was writing in Java at that time, so I had to juggle with a lot of things to get them working just right, but still…

Anyway, my project now is in Perl so I had to look a little bit around for any existing frameworks that might make my life easier, when I came up with the DANCER framework. And it was a brand new day…

That was two days ago. In this time I’ve been able to setup almost everything writing only a fraction of the code I used to. Now I am working on the particulars of the application behind the web-service. The setup of the service is straightforward; the DANCER dev team actually provides a tool to help along the way. But (as in every transition between languages) there are several bumps along the road after that. So, in order to help anyone interested, I’ll share any “recipes” that I come up along

Good coding to all!

File Upload using Perl and DANCER

Wed, 31 Jul 2013 00:00:00 +0300

One of the most common functions in a web application is File Upload. The following is a working code that can be used to this end. The prerequisites are only Perl (obviously) and the DANCER framework. Right! Here goes:

post '/upload/:file' =&gt; sub {
  my $upload_dir = "/home/fpsom/myApp/UPLOADS";
  my $filename = params-&gt;{file};
  my $uploadedFile = upload('file_input_foo');
  $uploadedFile-&gt;copy_to("$upload_dir/$filename");
  debug "My Log 1: " . params-&gt;{file};
  debug "My Log 2: " . ref($uploadedFile);
};

This should be copied in the myApp/lib/myApp.pm file (check for the corresponding file in your app)

In order to check the functionality of this code, I’ve used cURL as follows, where testUploadFile is an existing file:

curl -i -F file_input_foo=@testUploadFile http://localhost:3000/upload/testUploadFile

The output on the “development dance floor” should be as follows:

[10269]  core @0.000119&gt; request: POST /upload/testUploadFile from 127.0.0.1 in /usr/local/share/perl/5.14.2/Dancer/Handler.pm l. 56
[10269]  core @0.000413&gt; [hit #4]Trying to match 'POST /upload/testUploadFile' against /^\/upload\/([^\/]+)$/ (generated from '/upload/:file') in /usr/local/share/perl/5.14.2/Dancer/Route.pm l. 84
[10269]  core @0.000530&gt; [hit #4]  --&gt; got 1 in /usr/local/share/perl/5.14.2/Dancer/Route.pm l. 102
[10269]  core @0.000627&gt; [hit #4]  --&gt; named tokens are: file in /usr/local/share/perl/5.14.2/Dancer/Route.pm l. 130
[10269] debug @0.001051&gt; [hit #4]My Log 1: testUploadFile in /home/fpsom/myApp/lib/myApp.pm l. 21
[10269] debug @0.001145&gt; [hit #4]My Log 2: Dancer::Request::Upload in /home/fpsom/myApp/lib/myApp.pm l. 22
[10269]  core @0.001446&gt; [hit #4]response: 200 in /usr/local/share/perl/5.14.2/Dancer/Handler.pm l. 179

And that is that! Hope it helps.

Cloud Computing and Bioinformatics

RDA: a global research data orchestra

Share this:

September 2016 – the final tally

Share this:

Teaching my first Software Carpentry Workshop

Share this:

Training for NGS data analysis using Chipster

Share this:

Integrating datasets for bioinformatics

Share this:

Future opportunities and trends for e-infrastructures and life sciences

Share this:

Perl DANCER framework

Share this:

File Upload using Perl and DANCER

Share this: